Oct 30 06:53:54.864542 ip-10-0-132-173 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 06:53:54.864555 ip-10-0-132-173 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 06:53:54.864565 ip-10-0-132-173 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 06:53:54.864982 ip-10-0-132-173 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 06:54:04.428414 ip-10-0-132-173 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 06:54:04.428432 ip-10-0-132-173 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot c3ed987c4d5c44a8b8a70edb9002a3b6 -- Oct 30 06:55:50.968560 ip-10-0-132-173 systemd[1]: Starting Kubernetes Kubelet... Oct 30 06:55:51.364106 ip-10-0-132-173 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 06:55:51.364106 ip-10-0-132-173 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 06:55:51.364106 ip-10-0-132-173 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 06:55:51.364106 ip-10-0-132-173 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 06:55:51.368824 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368732 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:55:51.368824 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368823 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:55:51.368824 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368829 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:55:51.368824 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368832 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:55:51.368943 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368835 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:55:51.368943 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368838 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:55:51.368943 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368841 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:55:51.368943 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368844 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:55:51.368943 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368846 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:55:51.368943 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368849 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:55:51.368943 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368857 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:55:51.368943 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368861 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:55:51.368943 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368864 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:55:51.368943 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368866 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:55:51.368943 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368869 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:55:51.368943 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368871 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:55:51.368943 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368874 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:55:51.368943 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368877 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:55:51.368943 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368879 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:55:51.368943 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368882 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:55:51.368943 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368885 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:55:51.368943 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368887 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:55:51.368943 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368890 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:55:51.368943 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368893 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:55:51.369420 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368895 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:55:51.369420 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368898 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:55:51.369420 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368900 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:55:51.369420 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368903 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:55:51.369420 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368906 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:55:51.369420 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368908 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:55:51.369420 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368910 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:55:51.369420 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368913 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:55:51.369420 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368916 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:55:51.369420 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368919 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:55:51.369420 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368921 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:55:51.369420 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368923 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:55:51.369420 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368927 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:55:51.369420 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368930 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:55:51.369420 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368935 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:55:51.369420 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368940 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:55:51.369420 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368944 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:55:51.369420 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368947 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:55:51.369420 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368950 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:55:51.369862 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368954 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:55:51.369862 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368956 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:55:51.369862 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368959 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:55:51.369862 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368962 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:55:51.369862 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368965 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:55:51.369862 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368967 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:55:51.369862 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368970 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:55:51.369862 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368972 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:55:51.369862 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368975 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:55:51.369862 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368977 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:55:51.369862 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368980 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:55:51.369862 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368982 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:55:51.369862 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368985 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:55:51.369862 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368988 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:55:51.369862 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368990 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:55:51.369862 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368993 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:55:51.369862 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368995 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:55:51.369862 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.368998 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:55:51.369862 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369001 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:55:51.369862 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369004 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:55:51.370354 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369006 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:55:51.370354 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369009 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:55:51.370354 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369012 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:55:51.370354 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369015 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:55:51.370354 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369017 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:55:51.370354 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369021 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:55:51.370354 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369025 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:55:51.370354 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369028 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:55:51.370354 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369030 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:55:51.370354 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369034 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:55:51.370354 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369036 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:55:51.370354 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369039 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:55:51.370354 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369041 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:55:51.370354 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369044 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:55:51.370354 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369046 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:55:51.370354 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369049 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:55:51.370354 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369051 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:55:51.370354 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369054 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:55:51.370354 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369057 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:55:51.370354 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369059 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:55:51.370855 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369062 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:55:51.370855 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369067 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:55:51.370855 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369071 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:55:51.370855 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369074 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:55:51.370855 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369077 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:55:51.370855 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369080 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:55:51.370855 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369083 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:55:51.370855 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369086 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:55:51.370855 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369088 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:55:51.370855 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369091 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:55:51.370855 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369094 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:55:51.370855 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369096 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:55:51.370855 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369099 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:55:51.370855 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369101 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:55:51.370855 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369103 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:55:51.370855 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369106 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:55:51.370855 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369108 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:55:51.370855 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369111 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:55:51.370855 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369115 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:55:51.370855 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369117 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:55:51.371343 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369120 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:55:51.371343 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369122 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:55:51.371343 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369125 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:55:51.371343 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369965 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:55:51.371343 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369972 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:55:51.371343 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369975 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:55:51.371343 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369979 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:55:51.371343 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369982 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:55:51.371343 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369985 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:55:51.371343 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369987 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:55:51.371343 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369990 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:55:51.371343 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369993 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:55:51.371343 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369995 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:55:51.371343 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.369998 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:55:51.371343 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370001 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:55:51.371343 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370004 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:55:51.371343 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370006 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:55:51.371343 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370009 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:55:51.371343 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370011 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:55:51.371816 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370014 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:55:51.371816 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370017 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:55:51.371816 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370019 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:55:51.371816 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370022 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:55:51.371816 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370025 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:55:51.371816 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370027 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:55:51.371816 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370030 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:55:51.371816 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370032 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:55:51.371816 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370035 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:55:51.371816 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370037 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:55:51.371816 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370040 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:55:51.371816 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370043 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:55:51.371816 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370046 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:55:51.371816 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370048 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:55:51.371816 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370050 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:55:51.371816 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370053 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:55:51.371816 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370057 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:55:51.371816 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370059 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:55:51.371816 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370062 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:55:51.371816 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370065 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:55:51.372306 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370067 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:55:51.372306 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370070 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:55:51.372306 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370073 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:55:51.372306 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370076 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:55:51.372306 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370078 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:55:51.372306 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370081 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:55:51.372306 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370083 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:55:51.372306 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370086 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:55:51.372306 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370089 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:55:51.372306 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370091 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:55:51.372306 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370095 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:55:51.372306 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370099 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:55:51.372306 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370102 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:55:51.372306 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370106 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:55:51.372306 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370109 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:55:51.372306 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370112 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:55:51.372306 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370114 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:55:51.372306 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370117 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:55:51.372306 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370119 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:55:51.372849 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370122 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:55:51.372849 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370125 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:55:51.372849 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370127 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:55:51.372849 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370130 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:55:51.372849 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370132 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:55:51.372849 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370135 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:55:51.372849 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370137 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:55:51.372849 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370140 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:55:51.372849 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370142 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:55:51.372849 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370145 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:55:51.372849 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370149 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:55:51.372849 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370152 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:55:51.372849 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370154 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:55:51.372849 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370157 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:55:51.372849 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370161 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:55:51.372849 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370165 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:55:51.372849 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370168 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:55:51.372849 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370171 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:55:51.372849 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370174 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:55:51.373321 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370177 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:55:51.373321 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370182 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:55:51.373321 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370184 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:55:51.373321 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370187 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:55:51.373321 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370189 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:55:51.373321 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370192 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:55:51.373321 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370195 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:55:51.373321 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370197 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:55:51.373321 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370200 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:55:51.373321 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370202 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:55:51.373321 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370205 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:55:51.373321 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370207 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:55:51.373321 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370210 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:55:51.373321 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370213 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:55:51.373321 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370230 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:55:51.373321 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370233 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:55:51.373321 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370236 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:55:51.373321 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370238 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:55:51.373321 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370241 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:55:51.373801 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370244 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:55:51.373801 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370246 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:55:51.373801 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370249 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:55:51.373801 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370253 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:55:51.373801 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370256 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:55:51.373801 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370259 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:55:51.373801 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370262 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:55:51.373801 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370265 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:55:51.373801 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370267 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:55:51.373801 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370270 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:55:51.373801 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370273 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:55:51.373801 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370275 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:55:51.373801 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370278 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:55:51.373801 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370789 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:55:51.373801 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370795 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:55:51.373801 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370798 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:55:51.373801 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370800 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:55:51.373801 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370803 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:55:51.373801 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370806 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:55:51.373801 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370808 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:55:51.374298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370811 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:55:51.374298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370814 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:55:51.374298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370817 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:55:51.374298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370819 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:55:51.374298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370822 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:55:51.374298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370824 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:55:51.374298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370827 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:55:51.374298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370830 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:55:51.374298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370832 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:55:51.374298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370835 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:55:51.374298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370837 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:55:51.374298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370840 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:55:51.374298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370845 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:55:51.374298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370848 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:55:51.374298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370852 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:55:51.374298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370854 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:55:51.374298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370857 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:55:51.374298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370860 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:55:51.374298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370864 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:55:51.374745 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370866 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:55:51.374745 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370869 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:55:51.374745 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370874 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:55:51.374745 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370878 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:55:51.374745 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370881 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:55:51.374745 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370883 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:55:51.374745 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370886 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:55:51.374745 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370889 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:55:51.374745 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370891 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:55:51.374745 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370894 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:55:51.374745 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370896 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:55:51.374745 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370899 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:55:51.374745 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370901 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:55:51.374745 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370903 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:55:51.374745 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370906 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:55:51.374745 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370908 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:55:51.374745 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370911 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:55:51.374745 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370914 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:55:51.374745 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370916 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:55:51.375188 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370919 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:55:51.375188 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370921 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:55:51.375188 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370924 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:55:51.375188 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370926 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:55:51.375188 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370929 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:55:51.375188 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370931 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:55:51.375188 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370934 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:55:51.375188 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370936 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:55:51.375188 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370939 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:55:51.375188 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370942 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:55:51.375188 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370944 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:55:51.375188 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370947 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:55:51.375188 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370955 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:55:51.375188 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370958 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:55:51.375188 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370960 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:55:51.375188 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370963 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:55:51.375188 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370966 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:55:51.375188 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370968 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:55:51.375188 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370971 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:55:51.375188 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370974 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:55:51.375697 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370976 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:55:51.375697 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370980 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:55:51.375697 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370982 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:55:51.375697 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370985 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:55:51.375697 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370987 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:55:51.375697 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370989 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:55:51.375697 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370992 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:55:51.375697 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370995 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:55:51.375697 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.370997 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:55:51.375697 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.371000 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:55:51.375697 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.371002 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:55:51.375697 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.371005 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:55:51.375697 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.371007 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:55:51.375697 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.371010 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:55:51.375697 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.371012 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:55:51.375697 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.371015 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:55:51.375697 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.371017 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:55:51.375697 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.371020 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:55:51.375697 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.371022 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:55:51.375697 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.371025 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:55:51.376156 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.371028 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:55:51.376156 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.371030 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:55:51.376156 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.371032 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:55:51.376156 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.371035 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:55:51.376156 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.371038 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:55:51.376156 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.371047 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:55:51.376156 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.371050 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:55:51.376156 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.371053 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:55:51.376156 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.371056 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:55:51.376156 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.371058 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:55:51.376156 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.371061 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:55:51.376156 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.371063 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:55:51.376156 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.371066 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:55:51.376156 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.371069 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:55:51.376156 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.371071 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:55:51.376156 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.371074 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:55:51.376156 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.371077 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:55:51.376156 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.371079 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:55:51.376156 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.371082 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:55:51.376605 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.371086 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:55:51.376605 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.371089 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:55:51.376605 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372028 2574 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 06:55:51.376605 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372039 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 06:55:51.376605 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372046 2574 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 06:55:51.376605 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372051 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 06:55:51.376605 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372056 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 06:55:51.376605 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372059 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 06:55:51.376605 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372064 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 06:55:51.376605 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372069 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 06:55:51.376605 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372072 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 06:55:51.376605 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372075 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 06:55:51.376605 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372079 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 06:55:51.376605 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372083 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 06:55:51.376605 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372087 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 06:55:51.376605 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372090 2574 flags.go:64] FLAG: --cgroup-root="" Oct 30 06:55:51.376605 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372093 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 06:55:51.376605 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372096 2574 flags.go:64] FLAG: --client-ca-file="" Oct 30 06:55:51.376605 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372099 2574 flags.go:64] FLAG: --cloud-provider="external" Oct 30 06:55:51.376605 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372106 2574 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 06:55:51.376605 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372111 2574 flags.go:64] FLAG: --cluster-domain="" Oct 30 06:55:51.376605 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372114 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 06:55:51.376605 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372117 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 06:55:51.376605 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372120 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 06:55:51.377155 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372124 2574 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 06:55:51.377155 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372128 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 06:55:51.377155 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372131 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 06:55:51.377155 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372135 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 06:55:51.377155 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372139 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 06:55:51.377155 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372142 2574 flags.go:64] FLAG: --contention-profiling="false" Oct 30 06:55:51.377155 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372145 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 06:55:51.377155 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372148 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 06:55:51.377155 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372152 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 06:55:51.377155 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372155 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 06:55:51.377155 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372160 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 06:55:51.377155 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372163 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 06:55:51.377155 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372166 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 06:55:51.377155 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372169 2574 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 06:55:51.377155 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372172 2574 flags.go:64] FLAG: --enable-server="true" Oct 30 06:55:51.377155 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372176 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 06:55:51.377155 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372180 2574 flags.go:64] FLAG: --event-burst="100" Oct 30 06:55:51.377155 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372183 2574 flags.go:64] FLAG: --event-qps="50" Oct 30 06:55:51.377155 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372186 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 06:55:51.377155 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372190 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 06:55:51.377155 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372193 2574 flags.go:64] FLAG: --eviction-hard="" Oct 30 06:55:51.377155 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372198 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 06:55:51.377155 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372202 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 06:55:51.377155 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372205 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 06:55:51.377155 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372208 2574 flags.go:64] FLAG: --eviction-soft="" Oct 30 06:55:51.377795 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372211 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 06:55:51.377795 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372224 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 06:55:51.377795 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372230 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 06:55:51.377795 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372234 2574 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 06:55:51.377795 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372237 2574 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 06:55:51.377795 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372240 2574 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 06:55:51.377795 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372243 2574 flags.go:64] FLAG: --feature-gates="" Oct 30 06:55:51.377795 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372247 2574 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 06:55:51.377795 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372250 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 06:55:51.377795 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372254 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 06:55:51.377795 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372258 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 06:55:51.377795 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372261 2574 flags.go:64] FLAG: --healthz-port="10248" Oct 30 06:55:51.377795 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372264 2574 flags.go:64] FLAG: --help="false" Oct 30 06:55:51.377795 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372267 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-132-173.ec2.internal" Oct 30 06:55:51.377795 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372270 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 06:55:51.377795 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372273 2574 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 06:55:51.377795 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372276 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 06:55:51.377795 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372280 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 06:55:51.377795 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372283 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 06:55:51.377795 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372286 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 06:55:51.377795 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372290 2574 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 06:55:51.377795 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372293 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 06:55:51.377795 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372296 2574 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 06:55:51.378393 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372299 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 06:55:51.378393 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372302 2574 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 06:55:51.378393 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372305 2574 flags.go:64] FLAG: --kube-reserved="" Oct 30 06:55:51.378393 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372309 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 06:55:51.378393 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372312 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 06:55:51.378393 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372315 2574 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 06:55:51.378393 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372318 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 06:55:51.378393 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372321 2574 flags.go:64] FLAG: --lock-file="" Oct 30 06:55:51.378393 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372324 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 06:55:51.378393 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372327 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 06:55:51.378393 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372331 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 06:55:51.378393 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372337 2574 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 06:55:51.378393 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372340 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 06:55:51.378393 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372344 2574 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 06:55:51.378393 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372347 2574 flags.go:64] FLAG: --logging-format="text" Oct 30 06:55:51.378393 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372350 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 06:55:51.378393 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372353 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 06:55:51.378393 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372356 2574 flags.go:64] FLAG: --manifest-url="" Oct 30 06:55:51.378393 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372359 2574 flags.go:64] FLAG: --manifest-url-header="" Oct 30 06:55:51.378393 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372369 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 06:55:51.378393 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372372 2574 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 06:55:51.378393 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372380 2574 flags.go:64] FLAG: --max-pods="110" Oct 30 06:55:51.378393 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372383 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 06:55:51.378393 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372386 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 06:55:51.378393 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372389 2574 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 06:55:51.379416 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372392 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 06:55:51.379416 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372395 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 06:55:51.379416 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372399 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 06:55:51.379416 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372402 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 06:55:51.379416 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372410 2574 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 06:55:51.379416 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372413 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 06:55:51.379416 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372416 2574 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 06:55:51.379416 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372419 2574 flags.go:64] FLAG: --pod-cidr="" Oct 30 06:55:51.379416 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372422 2574 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 06:55:51.379416 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372425 2574 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 06:55:51.379416 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372428 2574 flags.go:64] FLAG: --pods-per-core="0" Oct 30 06:55:51.379416 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372431 2574 flags.go:64] FLAG: --port="10250" Oct 30 06:55:51.379416 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372434 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 06:55:51.379416 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372436 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-088610d53ff64ae41" Oct 30 06:55:51.379416 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372440 2574 flags.go:64] FLAG: --qos-reserved="" Oct 30 06:55:51.379416 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372443 2574 flags.go:64] FLAG: --read-only-port="10255" Oct 30 06:55:51.379416 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372446 2574 flags.go:64] FLAG: --register-node="true" Oct 30 06:55:51.379416 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372449 2574 flags.go:64] FLAG: --register-with-taints="" Oct 30 06:55:51.379416 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372453 2574 flags.go:64] FLAG: --registry-burst="10" Oct 30 06:55:51.379416 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372456 2574 flags.go:64] FLAG: --registry-qps="5" Oct 30 06:55:51.379416 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372459 2574 flags.go:64] FLAG: --reserved-cpus="" Oct 30 06:55:51.379416 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372462 2574 flags.go:64] FLAG: --reserved-memory="" Oct 30 06:55:51.379416 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372467 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 06:55:51.379416 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372470 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 06:55:51.379416 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372473 2574 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 06:55:51.379416 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372476 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 06:55:51.380524 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372479 2574 flags.go:64] FLAG: --runonce="false" Oct 30 06:55:51.380524 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372482 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 06:55:51.380524 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372485 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 06:55:51.380524 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372488 2574 flags.go:64] FLAG: --seccomp-default="false" Oct 30 06:55:51.380524 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372492 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 06:55:51.380524 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372495 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 06:55:51.380524 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372498 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 06:55:51.380524 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372501 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 06:55:51.380524 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372505 2574 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 06:55:51.380524 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372509 2574 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 06:55:51.380524 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372512 2574 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 06:55:51.380524 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372515 2574 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 06:55:51.380524 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372518 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 06:55:51.380524 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372521 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 06:55:51.380524 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372524 2574 flags.go:64] FLAG: --system-cgroups="" Oct 30 06:55:51.380524 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372526 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 06:55:51.380524 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372532 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 06:55:51.380524 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372535 2574 flags.go:64] FLAG: --tls-cert-file="" Oct 30 06:55:51.380524 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372538 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 06:55:51.380524 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372543 2574 flags.go:64] FLAG: --tls-min-version="" Oct 30 06:55:51.380524 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372546 2574 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 06:55:51.380524 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372549 2574 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 06:55:51.380524 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372552 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 06:55:51.380524 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372556 2574 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 06:55:51.380524 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372560 2574 flags.go:64] FLAG: --v="2" Oct 30 06:55:51.381461 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372564 2574 flags.go:64] FLAG: --version="false" Oct 30 06:55:51.381461 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372569 2574 flags.go:64] FLAG: --vmodule="" Oct 30 06:55:51.381461 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372573 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 06:55:51.381461 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.372578 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 06:55:51.381461 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372720 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:55:51.381461 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372724 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:55:51.381461 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372727 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:55:51.381461 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372729 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:55:51.381461 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372732 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:55:51.381461 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372735 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:55:51.381461 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372737 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:55:51.381461 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372740 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:55:51.381461 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372743 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:55:51.381461 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372745 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:55:51.381461 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372748 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:55:51.381461 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372750 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:55:51.381461 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372753 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:55:51.381461 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372756 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:55:51.381461 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372759 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:55:51.381461 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372761 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:55:51.381461 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372764 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:55:51.382126 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372766 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:55:51.382126 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372769 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:55:51.382126 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372771 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:55:51.382126 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372774 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:55:51.382126 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372777 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:55:51.382126 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372779 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:55:51.382126 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372782 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:55:51.382126 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372784 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:55:51.382126 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372787 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:55:51.382126 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372791 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:55:51.382126 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372793 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:55:51.382126 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372796 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:55:51.382126 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372798 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:55:51.382126 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372801 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:55:51.382126 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372803 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:55:51.382126 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372806 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:55:51.382126 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372809 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:55:51.382126 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372812 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:55:51.382126 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372814 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:55:51.382126 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372817 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:55:51.382619 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372820 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:55:51.382619 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372823 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:55:51.382619 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372826 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:55:51.382619 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372828 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:55:51.382619 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372831 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:55:51.382619 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372833 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:55:51.382619 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372839 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:55:51.382619 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372842 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:55:51.382619 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372845 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:55:51.382619 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372848 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:55:51.382619 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372851 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:55:51.382619 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372854 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:55:51.382619 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372856 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:55:51.382619 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372861 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:55:51.382619 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372864 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:55:51.382619 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372868 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:55:51.382619 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372871 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:55:51.382619 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372874 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:55:51.383049 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372877 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:55:51.383049 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372880 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:55:51.383049 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372882 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:55:51.383049 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372886 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:55:51.383049 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372889 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:55:51.383049 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372892 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:55:51.383049 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372894 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:55:51.383049 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372897 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:55:51.383049 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372900 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:55:51.383049 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372902 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:55:51.383049 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372906 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:55:51.383049 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372908 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:55:51.383049 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372911 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:55:51.383049 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372914 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:55:51.383049 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372917 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:55:51.383049 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372920 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:55:51.383049 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372922 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:55:51.383049 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372925 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:55:51.383049 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372927 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:55:51.383049 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372930 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:55:51.383535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372936 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:55:51.383535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372938 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:55:51.383535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372941 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:55:51.383535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372943 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:55:51.383535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372946 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:55:51.383535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372948 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:55:51.383535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372951 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:55:51.383535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372954 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:55:51.383535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372956 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:55:51.383535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372959 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:55:51.383535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372961 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:55:51.383535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372964 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:55:51.383535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372967 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:55:51.383535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372969 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:55:51.383535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372972 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:55:51.383535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372976 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:55:51.383535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372979 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:55:51.383535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372981 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:55:51.383535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372984 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:55:51.383535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372986 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:55:51.384043 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372989 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:55:51.384043 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372991 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:55:51.384043 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372994 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:55:51.384043 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.372997 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:55:51.384043 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373000 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:55:51.384043 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373002 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:55:51.384043 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373005 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:55:51.384043 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373007 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:55:51.384043 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373010 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:55:51.384043 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373012 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:55:51.384043 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373015 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:55:51.384043 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.373044 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 06:55:51.384043 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373081 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:55:51.384043 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373086 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:55:51.384043 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373090 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:55:51.384043 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373092 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:55:51.384443 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373095 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:55:51.384443 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373098 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:55:51.384443 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373100 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:55:51.384443 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373103 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:55:51.384443 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373106 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:55:51.384443 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373108 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:55:51.384443 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373111 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:55:51.384443 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373113 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:55:51.384443 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373116 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:55:51.384443 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373119 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:55:51.384443 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373122 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:55:51.384443 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373126 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:55:51.384443 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373129 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:55:51.384443 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373131 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:55:51.384443 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373134 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:55:51.384443 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373136 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:55:51.384443 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373139 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:55:51.384443 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373141 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:55:51.384443 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373144 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:55:51.384443 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373147 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:55:51.384909 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373150 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:55:51.384909 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373152 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:55:51.384909 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373155 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:55:51.384909 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373157 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:55:51.384909 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373160 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:55:51.384909 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373162 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:55:51.384909 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373165 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:55:51.384909 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373167 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:55:51.384909 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373170 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:55:51.384909 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373174 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:55:51.384909 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373176 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:55:51.384909 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373179 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:55:51.384909 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373181 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:55:51.384909 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373184 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:55:51.384909 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373187 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:55:51.384909 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373190 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:55:51.384909 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373192 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:55:51.384909 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373195 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:55:51.384909 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373198 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:55:51.384909 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373200 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:55:51.385408 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373203 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:55:51.385408 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373205 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:55:51.385408 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373208 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:55:51.385408 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373212 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:55:51.385408 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373228 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:55:51.385408 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373231 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:55:51.385408 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373234 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:55:51.385408 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373237 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:55:51.385408 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373239 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:55:51.385408 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373242 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:55:51.385408 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373244 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:55:51.385408 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373247 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:55:51.385408 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373250 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:55:51.385408 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373253 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:55:51.385408 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373256 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:55:51.385408 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373258 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:55:51.385408 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373261 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:55:51.385408 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373264 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:55:51.385408 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373266 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:55:51.385408 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373269 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:55:51.386048 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373271 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:55:51.386048 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373275 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:55:51.386048 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373278 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:55:51.386048 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373281 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:55:51.386048 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373283 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:55:51.386048 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373286 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:55:51.386048 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373289 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:55:51.386048 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373291 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:55:51.386048 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373294 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:55:51.386048 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373297 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:55:51.386048 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373299 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:55:51.386048 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373302 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:55:51.386048 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373305 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:55:51.386048 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373307 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:55:51.386048 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373310 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:55:51.386048 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373317 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:55:51.386048 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373321 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:55:51.386048 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373324 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:55:51.386048 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373328 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:55:51.386578 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373331 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:55:51.386578 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373334 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:55:51.386578 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373336 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:55:51.386578 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373340 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:55:51.386578 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373344 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:55:51.386578 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373346 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:55:51.386578 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373349 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:55:51.386578 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373352 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:55:51.386578 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373355 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:55:51.386578 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373358 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:55:51.386578 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373360 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:55:51.386578 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373363 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:55:51.386578 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373365 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:55:51.386578 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373368 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:55:51.386578 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373372 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:55:51.386578 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373374 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:55:51.386578 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373377 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:55:51.386578 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373379 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:55:51.386578 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373382 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:55:51.387017 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373384 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:55:51.387017 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373387 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:55:51.387017 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373390 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:55:51.387017 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.373392 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:55:51.387017 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.382286 2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 06:55:51.387017 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.382302 2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 06:55:51.387017 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.382996 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 06:55:51.387017 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.383003 2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 06:55:51.387017 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383064 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:55:51.387017 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383070 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:55:51.387017 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383073 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:55:51.387017 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383076 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:55:51.387017 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383078 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:55:51.387017 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383081 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:55:51.387017 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383083 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:55:51.387017 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383086 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:55:51.387017 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383089 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:55:51.387017 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383092 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:55:51.387017 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383094 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:55:51.387475 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383097 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:55:51.387475 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383100 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:55:51.387475 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383102 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:55:51.387475 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383105 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:55:51.387475 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383108 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:55:51.387475 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383111 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:55:51.387475 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383114 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:55:51.387475 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383118 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:55:51.387475 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383121 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:55:51.387475 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383124 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:55:51.387475 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383126 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:55:51.387475 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383130 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:55:51.387475 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383133 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:55:51.387475 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383135 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:55:51.387475 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383138 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:55:51.387475 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383141 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:55:51.387475 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383143 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:55:51.387475 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383146 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:55:51.387475 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383149 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:55:51.387907 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383151 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:55:51.387907 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383154 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:55:51.387907 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383156 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:55:51.387907 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383160 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:55:51.387907 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383163 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:55:51.387907 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383165 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:55:51.387907 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383168 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:55:51.387907 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383170 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:55:51.387907 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383173 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:55:51.387907 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383175 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:55:51.387907 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383178 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:55:51.387907 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383180 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:55:51.387907 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383183 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:55:51.387907 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383186 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:55:51.387907 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383188 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:55:51.387907 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383191 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:55:51.387907 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383194 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:55:51.387907 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383196 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:55:51.387907 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383199 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:55:51.388404 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383202 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:55:51.388404 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383205 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:55:51.388404 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383208 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:55:51.388404 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383211 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:55:51.388404 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383213 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:55:51.388404 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383232 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:55:51.388404 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383237 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:55:51.388404 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383241 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:55:51.388404 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383245 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:55:51.388404 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383248 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:55:51.388404 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383251 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:55:51.388404 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383254 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:55:51.388404 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383256 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:55:51.388404 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383259 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:55:51.388404 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383262 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:55:51.388404 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383265 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:55:51.388404 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383268 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:55:51.388404 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383270 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:55:51.388404 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383273 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:55:51.388404 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383276 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:55:51.388859 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383278 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:55:51.388859 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383281 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:55:51.388859 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383283 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:55:51.388859 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383286 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:55:51.388859 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383288 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:55:51.388859 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383291 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:55:51.388859 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383293 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:55:51.388859 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383296 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:55:51.388859 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383298 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:55:51.388859 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383301 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:55:51.388859 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383305 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:55:51.388859 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383308 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:55:51.388859 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383310 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:55:51.388859 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383313 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:55:51.388859 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383315 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:55:51.388859 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383318 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:55:51.388859 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383321 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:55:51.388859 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383323 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:55:51.388859 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383326 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:55:51.388859 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383329 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:55:51.389486 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383331 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:55:51.389486 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383334 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:55:51.389486 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383337 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:55:51.389486 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383339 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:55:51.389486 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383342 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:55:51.389486 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383344 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:55:51.389486 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383347 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:55:51.389486 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383349 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:55:51.389486 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383352 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:55:51.389486 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383355 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:55:51.389486 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383357 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:55:51.389486 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383360 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:55:51.389486 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383362 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:55:51.389486 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383365 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:55:51.389486 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383367 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:55:51.389486 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383370 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:55:51.389486 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383372 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:55:51.389898 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.383400 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 06:55:51.389898 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383539 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:55:51.389898 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383544 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:55:51.389898 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383547 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:55:51.389898 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383550 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:55:51.389898 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383553 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:55:51.389898 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383557 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:55:51.389898 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383560 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:55:51.389898 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383563 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:55:51.389898 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383566 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:55:51.389898 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383569 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:55:51.389898 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383571 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:55:51.389898 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383574 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:55:51.389898 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383576 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:55:51.389898 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383579 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:55:51.389898 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383582 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:55:51.390319 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383584 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:55:51.390319 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383587 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:55:51.390319 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383589 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:55:51.390319 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383593 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:55:51.390319 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383597 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:55:51.390319 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383600 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:55:51.390319 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383604 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:55:51.390319 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383607 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:55:51.390319 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383609 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:55:51.390319 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383612 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:55:51.390319 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383615 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:55:51.390319 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383618 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:55:51.390319 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383622 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:55:51.390319 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383625 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:55:51.390319 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383628 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:55:51.390319 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383631 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:55:51.390319 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383634 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:55:51.390319 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383637 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:55:51.390739 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383646 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:55:51.390739 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383650 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:55:51.390739 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383652 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:55:51.390739 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383655 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:55:51.390739 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383658 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:55:51.390739 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383662 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:55:51.390739 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383664 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:55:51.390739 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383667 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:55:51.390739 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383670 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:55:51.390739 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383672 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:55:51.390739 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383675 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:55:51.390739 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383678 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:55:51.390739 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383680 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:55:51.390739 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383683 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:55:51.390739 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383685 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:55:51.390739 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383688 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:55:51.390739 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383690 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:55:51.390739 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383693 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:55:51.390739 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383695 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:55:51.390739 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383698 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:55:51.391282 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383701 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:55:51.391282 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383703 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:55:51.391282 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383706 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:55:51.391282 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383708 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:55:51.391282 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383711 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:55:51.391282 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383713 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:55:51.391282 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383716 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:55:51.391282 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383718 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:55:51.391282 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383721 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:55:51.391282 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383724 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:55:51.391282 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383727 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:55:51.391282 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383729 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:55:51.391282 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383732 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:55:51.391282 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383735 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:55:51.391282 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383737 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:55:51.391282 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383740 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:55:51.391282 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383742 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:55:51.391282 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383745 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:55:51.391282 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383749 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:55:51.391282 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383752 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:55:51.391827 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383754 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:55:51.391827 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383757 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:55:51.391827 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383759 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:55:51.391827 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383762 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:55:51.391827 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383765 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:55:51.391827 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383767 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:55:51.391827 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383770 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:55:51.391827 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383773 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:55:51.391827 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383775 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:55:51.391827 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383778 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:55:51.391827 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383780 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:55:51.391827 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383783 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:55:51.391827 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383786 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:55:51.391827 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383788 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:55:51.391827 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383791 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:55:51.391827 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383793 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:55:51.391827 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383796 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:55:51.391827 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383798 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:55:51.391827 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383801 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:55:51.392298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383803 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:55:51.392298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383806 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:55:51.392298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383808 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:55:51.392298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383811 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:55:51.392298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383814 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:55:51.392298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383816 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:55:51.392298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383819 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:55:51.392298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383821 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:55:51.392298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383824 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:55:51.392298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383826 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:55:51.392298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383829 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:55:51.392298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383831 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:55:51.392298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383834 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:55:51.392298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383837 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:55:51.392298 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.383860 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 06:55:51.392298 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383891 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:55:51.392747 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383895 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:55:51.392747 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383898 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:55:51.392747 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383901 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:55:51.392747 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383904 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:55:51.392747 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383906 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:55:51.392747 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383909 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:55:51.392747 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383912 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:55:51.392747 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383914 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:55:51.392747 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383917 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:55:51.392747 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383922 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:55:51.392747 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383924 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:55:51.392747 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383927 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:55:51.392747 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383929 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:55:51.392747 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383932 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:55:51.392747 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383936 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:55:51.392747 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383939 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:55:51.392747 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383942 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:55:51.392747 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383945 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:55:51.392747 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383948 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:55:51.393523 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383950 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:55:51.393523 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383953 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:55:51.393523 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383956 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:55:51.393523 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383958 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:55:51.393523 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383961 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:55:51.393523 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383963 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:55:51.393523 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383966 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:55:51.393523 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383968 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:55:51.393523 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383972 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:55:51.393523 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383975 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:55:51.393523 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383978 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:55:51.393523 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383981 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:55:51.393523 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383984 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:55:51.393523 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383986 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:55:51.393523 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383989 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:55:51.393523 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383992 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:55:51.393523 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383995 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:55:51.393523 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.383999 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:55:51.393989 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384003 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:55:51.393989 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384006 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:55:51.393989 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384008 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:55:51.393989 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384011 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:55:51.393989 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384016 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:55:51.393989 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384018 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:55:51.393989 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384021 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:55:51.393989 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384024 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:55:51.393989 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384026 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:55:51.393989 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384030 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:55:51.393989 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384033 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:55:51.393989 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384035 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:55:51.393989 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384038 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:55:51.393989 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384041 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:55:51.393989 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384044 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:55:51.393989 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384047 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:55:51.393989 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384049 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:55:51.393989 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384052 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:55:51.393989 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384055 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:55:51.393989 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384057 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:55:51.394535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384060 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:55:51.394535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384063 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:55:51.394535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384065 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:55:51.394535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384068 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:55:51.394535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384071 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:55:51.394535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384074 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:55:51.394535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384077 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:55:51.394535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384079 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:55:51.394535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384082 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:55:51.394535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384084 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:55:51.394535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384087 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:55:51.394535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384089 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:55:51.394535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384092 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:55:51.394535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384094 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:55:51.394535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384098 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:55:51.394535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384101 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:55:51.394535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384104 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:55:51.394535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384107 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:55:51.394535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384109 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:55:51.394535 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384112 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:55:51.395273 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384115 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:55:51.395273 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384119 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:55:51.395273 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384123 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:55:51.395273 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384126 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:55:51.395273 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384128 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:55:51.395273 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384131 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:55:51.395273 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384134 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:55:51.395273 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384136 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:55:51.395273 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384138 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:55:51.395273 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384141 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:55:51.395273 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384144 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:55:51.395273 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384147 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:55:51.395273 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384149 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:55:51.395273 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384152 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:55:51.395273 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384155 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:55:51.395273 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384157 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:55:51.395273 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384160 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:55:51.395273 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384162 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:55:51.395273 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384165 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:55:51.395725 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384168 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:55:51.395725 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384171 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:55:51.395725 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384173 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:55:51.395725 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384176 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:55:51.395725 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384178 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:55:51.395725 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384180 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:55:51.395725 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384183 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:55:51.395725 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384187 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:55:51.395725 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:51.384189 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:55:51.395725 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.384262 2574 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 06:55:51.395725 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.387975 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 06:55:51.395725 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.388960 2574 server.go:1014] "Starting client certificate rotation" Oct 30 06:55:51.395725 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.389065 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 06:55:51.395725 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.389107 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 06:55:51.395725 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.390969 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 06:55:51.395725 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.392941 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 06:55:51.405452 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.405438 2574 log.go:25] "Validated CRI v1 runtime API" Oct 30 06:55:51.408624 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.408610 2574 log.go:25] "Validated CRI v1 image API" Oct 30 06:55:51.409405 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.409380 2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 06:55:51.414988 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.414971 2574 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 964a555e-b8df-45c2-a289-84189e81329e:/dev/nvme0n1p3 e464b85c-e9bd-468a-a7d7-26a80b3b8129:/dev/nvme0n1p4] Oct 30 06:55:51.415037 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.414988 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 06:55:51.421756 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.421644 2574 manager.go:217] Machine: {Timestamp:2026-10-30 06:55:51.420156673 +0000 UTC m=+0.334646327 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3092673 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26ddaaf9ec66d8c1a4f84ef88bc59c SystemUUID:ec26ddaa-f9ec-66d8-c1a4-f84ef88bc59c BootID:c3ed987c-4d5c-44a8-b8a7-0edb9002a3b6 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d6:0c:b7:c3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d6:0c:b7:c3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:47:e0:7a:ce:09 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 06:55:51.421833 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.421759 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 06:55:51.421875 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.421864 2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 06:55:51.423344 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.423326 2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 06:55:51.423478 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.423346 2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-173.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 06:55:51.423523 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.423481 2574 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 06:55:51.423523 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.423489 2574 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 06:55:51.423523 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.423519 2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 06:55:51.423594 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.423536 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 06:55:51.424343 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.424332 2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 06:55:51.427947 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.427936 2574 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 06:55:51.428055 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.428046 2574 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 06:55:51.428723 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.428714 2574 kubelet.go:493] "Attempting to sync node with API server" Oct 30 06:55:51.428758 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.428726 2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 06:55:51.428758 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.428736 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 06:55:51.428758 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.428747 2574 kubelet.go:405] "Adding apiserver pod source" Oct 30 06:55:51.428758 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.428754 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 06:55:51.429055 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.429044 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 06:55:51.432783 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.432765 2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 06:55:51.437439 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.437424 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 06:55:51.438163 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.438147 2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 06:55:51.438244 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.438180 2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 06:55:51.438369 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.438356 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 06:55:51.438428 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.438374 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 06:55:51.438428 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.438380 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 06:55:51.438428 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.438386 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 06:55:51.438428 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.438391 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 06:55:51.438428 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.438397 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 06:55:51.438988 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.438977 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 06:55:51.438988 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.438986 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 06:55:51.439062 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.438994 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 06:55:51.439062 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.438999 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 06:55:51.439062 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.439008 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 06:55:51.439062 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.439017 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 06:55:51.439705 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.439695 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 06:55:51.439747 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.439708 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 06:55:51.443450 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.443436 2574 server.go:1263] "Started kubelet" Oct 30 06:55:51.443567 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.443539 2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 06:55:51.443613 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.443537 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 06:55:51.443613 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.443603 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 06:55:51.444250 ip-10-0-132-173 systemd[1]: Started Kubernetes Kubelet. Oct 30 06:55:51.444882 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.444859 2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 06:55:51.445522 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.445508 2574 server.go:317] "Adding debug handlers to kubelet server" Oct 30 06:55:51.450579 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:51.450558 2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 06:55:51.452761 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.452744 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 06:55:51.453316 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.453290 2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 06:55:51.453419 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:51.453399 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 06:55:51.454047 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.454030 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 06:55:51.454047 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.454026 2574 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 06:55:51.454197 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.454060 2574 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 06:55:51.454197 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.454124 2574 reconstruct.go:97] "Volume reconstruction finished" Oct 30 06:55:51.454197 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.454128 2574 factory.go:55] Registering systemd factory Oct 30 06:55:51.454197 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.454151 2574 factory.go:223] Registration of the systemd container factory successfully Oct 30 06:55:51.454197 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.454131 2574 reconciler.go:29] "Reconciler: start to sync state" Oct 30 06:55:51.454614 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.454601 2574 factory.go:153] Registering CRI-O factory Oct 30 06:55:51.454614 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.454614 2574 factory.go:223] Registration of the crio container factory successfully Oct 30 06:55:51.454706 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.454655 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 06:55:51.454706 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.454667 2574 factory.go:103] Registering Raw factory Oct 30 06:55:51.454706 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.454678 2574 manager.go:1199] Started watching for new ooms in manager Oct 30 06:55:51.455592 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.455579 2574 manager.go:319] Starting recovery of all containers Oct 30 06:55:51.468518 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.468502 2574 manager.go:324] Recovery completed Oct 30 06:55:51.469587 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:51.469567 2574 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Oct 30 06:55:51.472733 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.472718 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 06:55:51.474477 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.474447 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientMemory" Oct 30 06:55:51.474540 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.474490 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 06:55:51.474540 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.474504 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientPID" Oct 30 06:55:51.475101 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.475088 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 06:55:51.475164 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.475086 2574 cpu_manager.go:226] "Starting" policy="none" Oct 30 06:55:51.475164 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.475156 2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 06:55:51.475233 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.475178 2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 06:55:51.475750 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.475734 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientMemory" Oct 30 06:55:51.475828 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.475758 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 06:55:51.475828 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.475767 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientPID" Oct 30 06:55:51.477455 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.477438 2574 policy_none.go:50] "Start" Oct 30 06:55:51.477455 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.477456 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 06:55:51.477541 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.477470 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 06:55:51.479458 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.479446 2574 policy_none.go:44] "Start" Oct 30 06:55:51.479895 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.479876 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 06:55:51.483386 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.483248 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-173.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 06:55:51.483386 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:51.483250 2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-173.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 06:55:51.484716 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:51.483290 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-173.ec2.internal.18e33c46cf17c6cf default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-173.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-173.ec2.internal,},FirstTimestamp:2026-10-30 06:55:51.443408591 +0000 UTC m=+0.357898244,LastTimestamp:2026-10-30 06:55:51.443408591 +0000 UTC m=+0.357898244,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-173.ec2.internal,}" Oct 30 06:55:51.501733 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:51.501709 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-173.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 06:55:51.501820 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:51.501694 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-173.ec2.internal.18e33c46d0f1c7da default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-173.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-173.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-173.ec2.internal,},FirstTimestamp:2026-10-30 06:55:51.474472922 +0000 UTC m=+0.388962576,LastTimestamp:2026-10-30 06:55:51.474472922 +0000 UTC m=+0.388962576,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-173.ec2.internal,}" Oct 30 06:55:51.501892 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:51.501867 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 06:55:51.501959 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:51.501939 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 06:55:51.507229 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.507202 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ln27c" Oct 30 06:55:51.512781 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.512758 2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 06:55:51.515114 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.515099 2574 manager.go:341] "Starting Device Plugin manager" Oct 30 06:55:51.515193 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:51.515130 2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 06:55:51.515193 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.515141 2574 server.go:87] "Starting device plugin registration server" Oct 30 06:55:51.515412 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.515398 2574 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 06:55:51.515456 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.515415 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 06:55:51.515668 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.515527 2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 06:55:51.515668 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.515609 2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 06:55:51.515668 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.515617 2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 06:55:51.516514 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:51.516492 2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 06:55:51.516609 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:51.516536 2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 06:55:51.523637 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.523619 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ln27c" Oct 30 06:55:51.528265 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:51.528183 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-173.ec2.internal.18e33c46d0f2281a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-173.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-173.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-173.ec2.internal,},FirstTimestamp:2026-10-30 06:55:51.474497562 +0000 UTC m=+0.388987216,LastTimestamp:2026-10-30 06:55:51.474497562 +0000 UTC m=+0.388987216,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-173.ec2.internal,}" Oct 30 06:55:51.582389 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.582365 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 06:55:51.583638 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.583612 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 06:55:51.583727 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.583648 2574 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 06:55:51.583727 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.583669 2574 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 06:55:51.583727 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:51.583712 2574 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 06:55:51.600228 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.600190 2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 06:55:51.615557 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.615492 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 06:55:51.616354 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.616338 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientMemory" Oct 30 06:55:51.616437 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.616366 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 06:55:51.616437 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.616375 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientPID" Oct 30 06:55:51.616437 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.616400 2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-173.ec2.internal" Oct 30 06:55:51.638544 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.638517 2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-173.ec2.internal" Oct 30 06:55:51.638544 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:51.638548 2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-173.ec2.internal\": node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 06:55:51.681435 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:51.681395 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 06:55:51.684697 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.684677 2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal"] Oct 30 06:55:51.684801 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.684731 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 06:55:51.685643 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.685616 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientMemory" Oct 30 06:55:51.685757 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.685653 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 06:55:51.685757 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.685667 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientPID" Oct 30 06:55:51.687615 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.687603 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 06:55:51.687778 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.687764 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 06:55:51.687830 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.687798 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 06:55:51.688304 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.688288 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientMemory" Oct 30 06:55:51.688377 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.688315 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 06:55:51.688377 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.688330 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientPID" Oct 30 06:55:51.688377 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.688340 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientMemory" Oct 30 06:55:51.688377 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.688357 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 06:55:51.688377 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.688369 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientPID" Oct 30 06:55:51.690228 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.690199 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" Oct 30 06:55:51.690277 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.690242 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 06:55:51.690934 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.690910 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientMemory" Oct 30 06:55:51.690934 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.690932 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 06:55:51.691021 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.690941 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientPID" Oct 30 06:55:51.718337 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:51.718318 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-173.ec2.internal\" not found" node="ip-10-0-132-173.ec2.internal" Oct 30 06:55:51.721225 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:51.721199 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-173.ec2.internal\" not found" node="ip-10-0-132-173.ec2.internal" Oct 30 06:55:51.781701 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:51.781671 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 06:55:51.856094 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.856046 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6ad70715366d9db6ff14ef77cdc50313-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal\" (UID: \"6ad70715366d9db6ff14ef77cdc50313\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 06:55:51.856094 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.856096 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ad70715366d9db6ff14ef77cdc50313-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal\" (UID: \"6ad70715366d9db6ff14ef77cdc50313\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 06:55:51.856352 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.856176 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b1dc20b8a7bd35e96a3f207cff8defc-config\") pod \"kube-apiserver-proxy-ip-10-0-132-173.ec2.internal\" (UID: \"5b1dc20b8a7bd35e96a3f207cff8defc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" Oct 30 06:55:51.882401 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:51.882335 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 06:55:51.956738 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.956671 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b1dc20b8a7bd35e96a3f207cff8defc-config\") pod \"kube-apiserver-proxy-ip-10-0-132-173.ec2.internal\" (UID: \"5b1dc20b8a7bd35e96a3f207cff8defc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" Oct 30 06:55:51.956738 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.956741 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6ad70715366d9db6ff14ef77cdc50313-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal\" (UID: \"6ad70715366d9db6ff14ef77cdc50313\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 06:55:51.956951 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.956768 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ad70715366d9db6ff14ef77cdc50313-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal\" (UID: \"6ad70715366d9db6ff14ef77cdc50313\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 06:55:51.956951 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.956804 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b1dc20b8a7bd35e96a3f207cff8defc-config\") pod \"kube-apiserver-proxy-ip-10-0-132-173.ec2.internal\" (UID: \"5b1dc20b8a7bd35e96a3f207cff8defc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" Oct 30 06:55:51.956951 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.956823 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ad70715366d9db6ff14ef77cdc50313-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal\" (UID: \"6ad70715366d9db6ff14ef77cdc50313\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 06:55:51.956951 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:51.956852 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6ad70715366d9db6ff14ef77cdc50313-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal\" (UID: \"6ad70715366d9db6ff14ef77cdc50313\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 06:55:51.983403 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:51.983337 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 06:55:52.022134 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:52.022103 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 06:55:52.024227 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:52.024203 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" Oct 30 06:55:52.083715 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:52.083670 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 06:55:52.184275 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:52.184181 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 06:55:54.070023 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:54.069992 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 06:55:54.170536 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:54.170504 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 06:55:54.173654 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:54.173635 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 06:55:54.173822 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:54.173797 2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 06:55:54.271296 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:54.271139 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 06:55:54.314506 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:54.314474 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 06:50:53 +0000 UTC" deadline="2028-03-31 08:19:55.178079447 +0000 UTC" Oct 30 06:55:54.314506 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:54.314502 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12433h24m0.863580239s" Oct 30 06:55:54.369926 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:54.369889 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad70715366d9db6ff14ef77cdc50313.slice/crio-749c81ba05deb3f15874314ed098daaf1fcede80c2304422e43619f8e5253292 WatchSource:0}: Error finding container 749c81ba05deb3f15874314ed098daaf1fcede80c2304422e43619f8e5253292: Status 404 returned error can't find the container with id 749c81ba05deb3f15874314ed098daaf1fcede80c2304422e43619f8e5253292 Oct 30 06:55:54.370278 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:54.370257 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b1dc20b8a7bd35e96a3f207cff8defc.slice/crio-b3c9bd7341cd590db1a41a04423e023234070dc4420f5a31c011e07d736ac772 WatchSource:0}: Error finding container b3c9bd7341cd590db1a41a04423e023234070dc4420f5a31c011e07d736ac772: Status 404 returned error can't find the container with id b3c9bd7341cd590db1a41a04423e023234070dc4420f5a31c011e07d736ac772 Oct 30 06:55:54.371866 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:54.371842 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 06:55:54.375745 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:54.375728 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:55:54.379680 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:54.379660 2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 06:55:54.386716 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:54.386692 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 06:55:54.388738 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:54.388721 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientMemory" Oct 30 06:55:54.388823 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:54.388751 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 06:55:54.388823 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:54.388760 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientPID" Oct 30 06:55:54.472111 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:54.472065 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 06:55:54.503978 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:54.503957 2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 06:55:54.572414 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:54.572371 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 06:55:54.672881 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:54.672847 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 06:55:54.773811 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:54.773723 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 06:55:54.874260 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:54.874226 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 06:55:54.974849 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:54.974811 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 06:55:55.075371 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:55.075297 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 06:55:55.140441 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.140409 2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 06:55:55.175731 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:55.175702 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 06:55:55.189924 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.189896 2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 06:55:55.215966 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.215941 2574 apiserver.go:52] "Watching apiserver" Oct 30 06:55:55.239656 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.239627 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 06:55:55.239656 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.239648 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" Oct 30 06:55:55.256962 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.256934 2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 06:55:55.257230 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.257212 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-t4rnr","openshift-ovn-kubernetes/ovnkube-node-8tlbb","kube-system/global-pull-secret-syncer-ts77g","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4","openshift-cluster-node-tuning-operator/tuned-qqnmt","openshift-dns/node-resolver-qq6f5","openshift-network-diagnostics/network-check-target-rjdmc","openshift-network-operator/iptables-alerter-zzvr7","kube-system/konnectivity-agent-mjwrd","openshift-image-registry/node-ca-9fpvq","openshift-multus/multus-q9jl6","openshift-multus/network-metrics-daemon-g6vbv"] Oct 30 06:55:55.259796 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.259770 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:55:55.262059 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.262038 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 06:55:55.262164 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.262102 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 06:55:55.262320 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.262304 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.264415 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.264394 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:55:55.264516 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:55.264481 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts77g" podUID="96024683-dd4e-4d78-b5ad-6aa9ccc4bc05" Oct 30 06:55:55.267476 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.266864 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:55:55.270253 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.270230 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.272532 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.272515 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qq6f5" Oct 30 06:55:55.275691 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.274834 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7vgdh\"" Oct 30 06:55:55.275691 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.275416 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 06:55:55.275691 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.275468 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 06:55:55.275691 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.275508 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 06:55:55.275691 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.275542 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 06:55:55.275691 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.275680 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 06:55:55.277125 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.277107 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjdmc" Oct 30 06:55:55.277210 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:55.277164 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjdmc" podUID="16e1a28b-a156-4d4e-820f-8712616b4eb2" Oct 30 06:55:55.279373 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.279357 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zzvr7" Oct 30 06:55:55.281375 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.281343 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mjwrd" Oct 30 06:55:55.284921 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.281832 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 06:55:55.284921 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.282077 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fvc75\"" Oct 30 06:55:55.284921 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.283266 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 06:55:55.285182 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.285100 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9fpvq" Oct 30 06:55:55.287684 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.287666 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 06:55:55.288547 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.288528 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fp7lr\"" Oct 30 06:55:55.288690 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.288674 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.290549 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.290530 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal"] Oct 30 06:55:55.290702 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.290555 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal"] Oct 30 06:55:55.290702 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.290641 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:55:55.290702 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:55.290689 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6vbv" podUID="f1b14112-6c97-4bfa-8f2a-5988f59ad06a" Oct 30 06:55:55.294292 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.294267 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 06:55:55.294385 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.294337 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tlwcv" Oct 30 06:55:55.305719 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.305698 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tlwcv" Oct 30 06:55:55.341153 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.341133 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 06:55:55.352155 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.352133 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-sysctl-conf\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.352278 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.352181 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-sys\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.352278 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.352207 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-cni-binary-copy\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.352386 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.352291 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-run-k8s-cni-cncf-io\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.352386 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.352320 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-multus-daemon-config\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.352386 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.352342 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-slash\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.352386 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.352359 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-socket-dir\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:55:55.352675 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.352398 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d1b84f62-7f09-42d0-8ed1-a2447efa3bf3-serviceca\") pod \"node-ca-9fpvq\" (UID: \"d1b84f62-7f09-42d0-8ed1-a2447efa3bf3\") " pod="openshift-image-registry/node-ca-9fpvq" Oct 30 06:55:55.352675 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.352458 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlg4z\" (UniqueName: \"kubernetes.io/projected/d1b84f62-7f09-42d0-8ed1-a2447efa3bf3-kube-api-access-tlg4z\") pod \"node-ca-9fpvq\" (UID: \"d1b84f62-7f09-42d0-8ed1-a2447efa3bf3\") " pod="openshift-image-registry/node-ca-9fpvq" Oct 30 06:55:55.352675 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.352500 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-modprobe-d\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.352675 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.352527 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-sysconfig\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.352675 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.352551 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-multus-conf-dir\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.352675 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.352620 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-kubelet-config\") pod \"global-pull-secret-syncer-ts77g\" (UID: \"96024683-dd4e-4d78-b5ad-6aa9ccc4bc05\") " pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:55:55.352675 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.352653 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-hostroot\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.352675 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.352670 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-cni-binary-copy\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:55:55.353056 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.352700 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-systemd-units\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.353056 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.352734 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-var-lib-openvswitch\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.353056 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.352756 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-ovnkube-config\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.353056 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.352807 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-env-overrides\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.353056 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.352862 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-system-cni-dir\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:55:55.353056 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.352886 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:55:55.353056 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.352918 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-ovnkube-script-lib\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.353056 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.352937 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-run-multus-certs\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.353056 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.352959 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-registration-dir\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:55:55.353056 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.352983 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-kubelet\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.353056 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353038 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-cni-netd\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.353553 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353063 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:55:55.353553 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353090 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:55:55.353553 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353110 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-dbus\") pod \"global-pull-secret-syncer-ts77g\" (UID: \"96024683-dd4e-4d78-b5ad-6aa9ccc4bc05\") " pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:55:55.353553 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353134 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-metrics-certs\") pod \"network-metrics-daemon-g6vbv\" (UID: \"f1b14112-6c97-4bfa-8f2a-5988f59ad06a\") " pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:55:55.353553 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353162 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-sys-fs\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:55:55.353553 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353177 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bbda0b6e-80fb-4964-a0a9-23c69a84889e-host-slash\") pod \"iptables-alerter-zzvr7\" (UID: \"bbda0b6e-80fb-4964-a0a9-23c69a84889e\") " pod="openshift-network-operator/iptables-alerter-zzvr7" Oct 30 06:55:55.353553 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353197 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-etc-openvswitch\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.353553 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353267 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-lib-modules\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.353553 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353293 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-tuned\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.353553 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353316 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9jzs\" (UniqueName: \"kubernetes.io/projected/0994513f-0dac-4106-8bfe-5a89cbc11f02-kube-api-access-x9jzs\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.353553 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353331 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4664d551-2cbf-42c5-a730-0d8c088f4f0f-hosts-file\") pod \"node-resolver-qq6f5\" (UID: \"4664d551-2cbf-42c5-a730-0d8c088f4f0f\") " pod="openshift-dns/node-resolver-qq6f5" Oct 30 06:55:55.353553 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353351 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-original-pull-secret\") pod \"global-pull-secret-syncer-ts77g\" (UID: \"96024683-dd4e-4d78-b5ad-6aa9ccc4bc05\") " pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:55:55.353553 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353366 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-cni-bin\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.353553 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353396 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-kubernetes\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.353553 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353410 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-cnibin\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.353553 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353428 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-run-systemd\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.354296 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353465 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-device-dir\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:55:55.354296 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353490 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bx54\" (UniqueName: \"kubernetes.io/projected/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-kube-api-access-2bx54\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:55:55.354296 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353514 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-node-log\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.354296 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353538 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-var-lib-kubelet\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.354296 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353563 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbp46\" (UniqueName: \"kubernetes.io/projected/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-kube-api-access-xbp46\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.354296 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353609 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:55:55.354296 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353710 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-run-netns\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.354296 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353735 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4664d551-2cbf-42c5-a730-0d8c088f4f0f-tmp-dir\") pod \"node-resolver-qq6f5\" (UID: \"4664d551-2cbf-42c5-a730-0d8c088f4f0f\") " pod="openshift-dns/node-resolver-qq6f5" Oct 30 06:55:55.354296 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353778 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-system-cni-dir\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.354296 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353804 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-run-netns\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.354296 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353824 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-var-lib-cni-bin\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.354296 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353839 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-etc-kubernetes\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.354296 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353860 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-run-openvswitch\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.354296 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353883 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-log-socket\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.354296 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353904 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-ovn-node-metrics-cert\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.354296 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353931 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l5w2\" (UniqueName: \"kubernetes.io/projected/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-kube-api-access-2l5w2\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.354296 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353947 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0994513f-0dac-4106-8bfe-5a89cbc11f02-tmp\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.355019 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353972 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5225f97b-21b7-453c-8d19-ccfce5ee6d54-agent-certs\") pod \"konnectivity-agent-mjwrd\" (UID: \"5225f97b-21b7-453c-8d19-ccfce5ee6d54\") " pod="kube-system/konnectivity-agent-mjwrd" Oct 30 06:55:55.355019 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.353997 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-var-lib-cni-multus\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.355019 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.354034 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bbda0b6e-80fb-4964-a0a9-23c69a84889e-iptables-alerter-script\") pod \"iptables-alerter-zzvr7\" (UID: \"bbda0b6e-80fb-4964-a0a9-23c69a84889e\") " pod="openshift-network-operator/iptables-alerter-zzvr7" Oct 30 06:55:55.355019 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.354060 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-sysctl-d\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.355019 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.354084 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-var-lib-kubelet\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.355019 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.354107 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-run-ovn\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.355019 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.354128 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-systemd\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.355019 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.354149 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-host\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.355019 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.354174 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5225f97b-21b7-453c-8d19-ccfce5ee6d54-konnectivity-ca\") pod \"konnectivity-agent-mjwrd\" (UID: \"5225f97b-21b7-453c-8d19-ccfce5ee6d54\") " pod="kube-system/konnectivity-agent-mjwrd" Oct 30 06:55:55.355019 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.354193 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-multus-cni-dir\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.355019 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.354213 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-etc-selinux\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:55:55.355019 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.354227 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-run\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.355019 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.354250 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6527\" (UniqueName: \"kubernetes.io/projected/4664d551-2cbf-42c5-a730-0d8c088f4f0f-kube-api-access-t6527\") pod \"node-resolver-qq6f5\" (UID: \"4664d551-2cbf-42c5-a730-0d8c088f4f0f\") " pod="openshift-dns/node-resolver-qq6f5" Oct 30 06:55:55.355019 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.354274 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-multus-socket-dir-parent\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.355019 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.354300 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v8q5\" (UniqueName: \"kubernetes.io/projected/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-kube-api-access-6v8q5\") pod \"network-metrics-daemon-g6vbv\" (UID: \"f1b14112-6c97-4bfa-8f2a-5988f59ad06a\") " pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:55:55.355019 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.354328 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7ndt\" (UniqueName: \"kubernetes.io/projected/16e1a28b-a156-4d4e-820f-8712616b4eb2-kube-api-access-x7ndt\") pod \"network-check-target-rjdmc\" (UID: \"16e1a28b-a156-4d4e-820f-8712616b4eb2\") " pod="openshift-network-diagnostics/network-check-target-rjdmc" Oct 30 06:55:55.355502 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.354355 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj7gf\" (UniqueName: \"kubernetes.io/projected/bbda0b6e-80fb-4964-a0a9-23c69a84889e-kube-api-access-zj7gf\") pod \"iptables-alerter-zzvr7\" (UID: \"bbda0b6e-80fb-4964-a0a9-23c69a84889e\") " pod="openshift-network-operator/iptables-alerter-zzvr7" Oct 30 06:55:55.355502 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.354378 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d1b84f62-7f09-42d0-8ed1-a2447efa3bf3-host\") pod \"node-ca-9fpvq\" (UID: \"d1b84f62-7f09-42d0-8ed1-a2447efa3bf3\") " pod="openshift-image-registry/node-ca-9fpvq" Oct 30 06:55:55.355502 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.354403 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.355502 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.354563 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-cnibin\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:55:55.355502 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.354600 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbzjd\" (UniqueName: \"kubernetes.io/projected/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-kube-api-access-fbzjd\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:55:55.355502 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.354623 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-run-ovn-kubernetes\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.373671 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.373612 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" event={"ID":"5b1dc20b8a7bd35e96a3f207cff8defc","Type":"ContainerStarted","Data":"b3c9bd7341cd590db1a41a04423e023234070dc4420f5a31c011e07d736ac772"} Oct 30 06:55:55.374778 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.374753 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" event={"ID":"6ad70715366d9db6ff14ef77cdc50313","Type":"ContainerStarted","Data":"749c81ba05deb3f15874314ed098daaf1fcede80c2304422e43619f8e5253292"} Oct 30 06:55:55.376399 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.376380 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 06:55:55.376492 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.376401 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 06:55:55.376492 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.376463 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 06:55:55.376564 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.376494 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 06:55:55.376564 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.376517 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 06:55:55.376564 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.376550 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 06:55:55.376691 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.376565 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 06:55:55.376691 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.376622 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 06:55:55.376797 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.376588 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 06:55:55.376968 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.376948 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 06:55:55.377073 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.376965 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 06:55:55.377073 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.377012 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 06:55:55.377073 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.377040 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 06:55:55.377073 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.377057 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7f8th\"" Oct 30 06:55:55.377073 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.377074 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cmb4r\"" Oct 30 06:55:55.377324 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.377096 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zl222\"" Oct 30 06:55:55.377324 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.377103 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 06:55:55.377324 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.377130 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hwx8p\"" Oct 30 06:55:55.377324 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.377138 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 06:55:55.377324 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.377145 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 06:55:55.377324 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.377189 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r8khf\"" Oct 30 06:55:55.377324 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.377192 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 06:55:55.377324 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.377203 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4hfgh\"" Oct 30 06:55:55.377324 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.377230 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 06:55:55.377324 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.377266 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 06:55:55.377324 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.377271 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 06:55:55.455121 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455086 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-sys-fs\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:55:55.455312 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455134 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bbda0b6e-80fb-4964-a0a9-23c69a84889e-host-slash\") pod \"iptables-alerter-zzvr7\" (UID: \"bbda0b6e-80fb-4964-a0a9-23c69a84889e\") " pod="openshift-network-operator/iptables-alerter-zzvr7" Oct 30 06:55:55.455312 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455164 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-etc-openvswitch\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.455312 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455189 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-lib-modules\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.455312 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455212 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-tuned\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.455312 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455235 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9jzs\" (UniqueName: \"kubernetes.io/projected/0994513f-0dac-4106-8bfe-5a89cbc11f02-kube-api-access-x9jzs\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.455312 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455257 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4664d551-2cbf-42c5-a730-0d8c088f4f0f-hosts-file\") pod \"node-resolver-qq6f5\" (UID: \"4664d551-2cbf-42c5-a730-0d8c088f4f0f\") " pod="openshift-dns/node-resolver-qq6f5" Oct 30 06:55:55.455312 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455284 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-original-pull-secret\") pod \"global-pull-secret-syncer-ts77g\" (UID: \"96024683-dd4e-4d78-b5ad-6aa9ccc4bc05\") " pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:55:55.455312 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455306 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-cni-bin\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.455720 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455329 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-kubernetes\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.455720 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455353 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-cnibin\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.455720 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455374 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-run-systemd\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.455720 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455400 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-device-dir\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:55:55.455720 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455416 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bx54\" (UniqueName: \"kubernetes.io/projected/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-kube-api-access-2bx54\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:55:55.455720 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455432 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-node-log\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.455720 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455447 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-var-lib-kubelet\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.455720 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455468 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbp46\" (UniqueName: \"kubernetes.io/projected/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-kube-api-access-xbp46\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.455720 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455489 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:55:55.455720 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455508 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-run-netns\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.455720 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455526 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4664d551-2cbf-42c5-a730-0d8c088f4f0f-tmp-dir\") pod \"node-resolver-qq6f5\" (UID: \"4664d551-2cbf-42c5-a730-0d8c088f4f0f\") " pod="openshift-dns/node-resolver-qq6f5" Oct 30 06:55:55.455720 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455540 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-system-cni-dir\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.455720 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455563 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-run-netns\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.455720 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455599 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-var-lib-cni-bin\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.455720 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455613 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-etc-kubernetes\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.455720 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455628 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-run-openvswitch\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.455720 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455643 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-log-socket\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.455720 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455657 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-ovn-node-metrics-cert\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.456257 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455678 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l5w2\" (UniqueName: \"kubernetes.io/projected/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-kube-api-access-2l5w2\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.456257 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455696 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0994513f-0dac-4106-8bfe-5a89cbc11f02-tmp\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.456257 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455730 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5225f97b-21b7-453c-8d19-ccfce5ee6d54-agent-certs\") pod \"konnectivity-agent-mjwrd\" (UID: \"5225f97b-21b7-453c-8d19-ccfce5ee6d54\") " pod="kube-system/konnectivity-agent-mjwrd" Oct 30 06:55:55.456257 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455750 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-var-lib-cni-multus\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.456257 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455772 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bbda0b6e-80fb-4964-a0a9-23c69a84889e-iptables-alerter-script\") pod \"iptables-alerter-zzvr7\" (UID: \"bbda0b6e-80fb-4964-a0a9-23c69a84889e\") " pod="openshift-network-operator/iptables-alerter-zzvr7" Oct 30 06:55:55.456257 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455787 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-sysctl-d\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.456257 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455801 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-var-lib-kubelet\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.456257 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455817 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-run-ovn\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.456257 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455841 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-systemd\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.456257 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455855 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-host\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.456257 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455869 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5225f97b-21b7-453c-8d19-ccfce5ee6d54-konnectivity-ca\") pod \"konnectivity-agent-mjwrd\" (UID: \"5225f97b-21b7-453c-8d19-ccfce5ee6d54\") " pod="kube-system/konnectivity-agent-mjwrd" Oct 30 06:55:55.456257 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455883 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-multus-cni-dir\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.456257 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455897 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-etc-selinux\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:55:55.456257 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455918 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-run\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.456257 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455934 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6527\" (UniqueName: \"kubernetes.io/projected/4664d551-2cbf-42c5-a730-0d8c088f4f0f-kube-api-access-t6527\") pod \"node-resolver-qq6f5\" (UID: \"4664d551-2cbf-42c5-a730-0d8c088f4f0f\") " pod="openshift-dns/node-resolver-qq6f5" Oct 30 06:55:55.456257 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455949 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-multus-socket-dir-parent\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.456257 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455964 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v8q5\" (UniqueName: \"kubernetes.io/projected/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-kube-api-access-6v8q5\") pod \"network-metrics-daemon-g6vbv\" (UID: \"f1b14112-6c97-4bfa-8f2a-5988f59ad06a\") " pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:55:55.456812 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.455987 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7ndt\" (UniqueName: \"kubernetes.io/projected/16e1a28b-a156-4d4e-820f-8712616b4eb2-kube-api-access-x7ndt\") pod \"network-check-target-rjdmc\" (UID: \"16e1a28b-a156-4d4e-820f-8712616b4eb2\") " pod="openshift-network-diagnostics/network-check-target-rjdmc" Oct 30 06:55:55.456812 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456006 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj7gf\" (UniqueName: \"kubernetes.io/projected/bbda0b6e-80fb-4964-a0a9-23c69a84889e-kube-api-access-zj7gf\") pod \"iptables-alerter-zzvr7\" (UID: \"bbda0b6e-80fb-4964-a0a9-23c69a84889e\") " pod="openshift-network-operator/iptables-alerter-zzvr7" Oct 30 06:55:55.456812 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456021 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d1b84f62-7f09-42d0-8ed1-a2447efa3bf3-host\") pod \"node-ca-9fpvq\" (UID: \"d1b84f62-7f09-42d0-8ed1-a2447efa3bf3\") " pod="openshift-image-registry/node-ca-9fpvq" Oct 30 06:55:55.456812 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456036 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.456812 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456050 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-cnibin\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:55:55.456812 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456071 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbzjd\" (UniqueName: \"kubernetes.io/projected/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-kube-api-access-fbzjd\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:55:55.456812 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456091 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-run-ovn-kubernetes\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.456812 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456105 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-sysctl-conf\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.456812 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456120 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-sys\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.456812 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456134 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-cni-binary-copy\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.456812 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456156 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-run-k8s-cni-cncf-io\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.456812 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456161 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4664d551-2cbf-42c5-a730-0d8c088f4f0f-tmp-dir\") pod \"node-resolver-qq6f5\" (UID: \"4664d551-2cbf-42c5-a730-0d8c088f4f0f\") " pod="openshift-dns/node-resolver-qq6f5" Oct 30 06:55:55.456812 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456172 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-multus-daemon-config\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.456812 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456187 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-slash\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.456812 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456202 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-socket-dir\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:55:55.456812 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456219 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d1b84f62-7f09-42d0-8ed1-a2447efa3bf3-serviceca\") pod \"node-ca-9fpvq\" (UID: \"d1b84f62-7f09-42d0-8ed1-a2447efa3bf3\") " pod="openshift-image-registry/node-ca-9fpvq" Oct 30 06:55:55.456812 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456230 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-sys-fs\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:55:55.457471 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456237 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlg4z\" (UniqueName: \"kubernetes.io/projected/d1b84f62-7f09-42d0-8ed1-a2447efa3bf3-kube-api-access-tlg4z\") pod \"node-ca-9fpvq\" (UID: \"d1b84f62-7f09-42d0-8ed1-a2447efa3bf3\") " pod="openshift-image-registry/node-ca-9fpvq" Oct 30 06:55:55.457471 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456262 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bbda0b6e-80fb-4964-a0a9-23c69a84889e-host-slash\") pod \"iptables-alerter-zzvr7\" (UID: \"bbda0b6e-80fb-4964-a0a9-23c69a84889e\") " pod="openshift-network-operator/iptables-alerter-zzvr7" Oct 30 06:55:55.457471 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456284 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-modprobe-d\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.457471 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456302 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-sysconfig\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.457471 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456318 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-multus-conf-dir\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.457471 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456338 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-kubelet-config\") pod \"global-pull-secret-syncer-ts77g\" (UID: \"96024683-dd4e-4d78-b5ad-6aa9ccc4bc05\") " pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:55:55.457471 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456353 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-hostroot\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.457471 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456369 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-cni-binary-copy\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:55:55.457471 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456388 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-systemd-units\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.457471 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456405 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-var-lib-openvswitch\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.457471 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456420 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-ovnkube-config\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.457471 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456435 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-env-overrides\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.457471 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456452 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-system-cni-dir\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:55:55.457471 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456472 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:55:55.457471 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456489 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-ovnkube-script-lib\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.457471 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456506 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-run-multus-certs\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.457471 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456525 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-registration-dir\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:55:55.458222 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456540 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-kubelet\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.458222 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456556 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-cni-netd\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.458222 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456586 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-system-cni-dir\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.458222 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456592 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:55:55.458222 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456619 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:55:55.458222 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456634 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-run-netns\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.458222 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456643 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-dbus\") pod \"global-pull-secret-syncer-ts77g\" (UID: \"96024683-dd4e-4d78-b5ad-6aa9ccc4bc05\") " pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:55:55.458222 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456650 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-run-netns\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.458222 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456669 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-metrics-certs\") pod \"network-metrics-daemon-g6vbv\" (UID: \"f1b14112-6c97-4bfa-8f2a-5988f59ad06a\") " pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:55:55.458222 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456677 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-kubernetes\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.458222 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456681 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-var-lib-cni-bin\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.458222 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456700 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-node-log\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.458222 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456715 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-etc-kubernetes\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.458222 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456756 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-run-openvswitch\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.458222 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456796 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4664d551-2cbf-42c5-a730-0d8c088f4f0f-hosts-file\") pod \"node-resolver-qq6f5\" (UID: \"4664d551-2cbf-42c5-a730-0d8c088f4f0f\") " pod="openshift-dns/node-resolver-qq6f5" Oct 30 06:55:55.458222 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456829 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-modprobe-d\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.458222 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456887 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-multus-conf-dir\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.458222 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456894 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-log-socket\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.458916 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456941 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-var-lib-kubelet\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.458916 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456991 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-cnibin\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.458916 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.457002 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-run-ovn\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.458916 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.457040 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-run-systemd\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.458916 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.457042 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-systemd\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.458916 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.457087 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-device-dir\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:55:55.458916 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.457115 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-host\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.458916 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.456763 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-cni-bin\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.458916 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.457362 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-sysctl-d\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.458916 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.457437 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-registration-dir\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:55:55.458916 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.457476 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-sys\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.458916 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.457492 2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 06:55:55.458916 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.457521 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bbda0b6e-80fb-4964-a0a9-23c69a84889e-iptables-alerter-script\") pod \"iptables-alerter-zzvr7\" (UID: \"bbda0b6e-80fb-4964-a0a9-23c69a84889e\") " pod="openshift-network-operator/iptables-alerter-zzvr7" Oct 30 06:55:55.458916 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.457534 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5225f97b-21b7-453c-8d19-ccfce5ee6d54-konnectivity-ca\") pod \"konnectivity-agent-mjwrd\" (UID: \"5225f97b-21b7-453c-8d19-ccfce5ee6d54\") " pod="kube-system/konnectivity-agent-mjwrd" Oct 30 06:55:55.458916 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.457537 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-kubelet-config\") pod \"global-pull-secret-syncer-ts77g\" (UID: \"96024683-dd4e-4d78-b5ad-6aa9ccc4bc05\") " pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:55:55.458916 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.457588 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-run\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.458916 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.457598 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-hostroot\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.458916 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.457618 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-etc-openvswitch\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.458916 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:55.457629 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 06:55:55.459655 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:55.457693 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-original-pull-secret podName:96024683-dd4e-4d78-b5ad-6aa9ccc4bc05 nodeName:}" failed. No retries permitted until 2026-10-30 06:55:55.957675084 +0000 UTC m=+3.086811594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-original-pull-secret") pod "global-pull-secret-syncer-ts77g" (UID: "96024683-dd4e-4d78-b5ad-6aa9ccc4bc05") : object "kube-system"/"original-pull-secret" not registered Oct 30 06:55:55.459655 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.457701 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-socket-dir\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:55:55.459655 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.457789 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-slash\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.459655 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.457494 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-etc-selinux\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:55:55.459655 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.457838 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-kubelet\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.459655 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.457864 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-multus-socket-dir-parent\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.459655 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.457901 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-var-lib-openvswitch\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.459655 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.458035 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-cni-netd\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.459655 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.458068 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-lib-modules\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.459655 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.458127 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-var-lib-kubelet\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.459655 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.458340 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d1b84f62-7f09-42d0-8ed1-a2447efa3bf3-host\") pod \"node-ca-9fpvq\" (UID: \"d1b84f62-7f09-42d0-8ed1-a2447efa3bf3\") " pod="openshift-image-registry/node-ca-9fpvq" Oct 30 06:55:55.459655 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.458366 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.459655 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.458489 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-run-ovn-kubernetes\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.459655 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.458495 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-var-lib-cni-multus\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.459655 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.458530 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-run-k8s-cni-cncf-io\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.459655 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.458596 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-sysctl-conf\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.459655 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.457871 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-systemd-units\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.460428 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.458657 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-run-multus-certs\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.460428 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:55.458729 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:55:55.460428 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:55.458776 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-metrics-certs podName:f1b14112-6c97-4bfa-8f2a-5988f59ad06a nodeName:}" failed. No retries permitted until 2026-10-30 06:55:55.958761021 +0000 UTC m=+3.087897526 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-metrics-certs") pod "network-metrics-daemon-g6vbv" (UID: "f1b14112-6c97-4bfa-8f2a-5988f59ad06a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:55:55.460428 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.458806 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:55:55.460428 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.458816 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-cnibin\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:55:55.460428 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.458885 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-multus-cni-dir\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.460428 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.458905 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-dbus\") pod \"global-pull-secret-syncer-ts77g\" (UID: \"96024683-dd4e-4d78-b5ad-6aa9ccc4bc05\") " pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:55:55.460428 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.458950 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-system-cni-dir\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:55:55.460428 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.458993 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-sysconfig\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.460428 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.459221 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-cni-binary-copy\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.460428 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.459489 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-cni-binary-copy\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:55:55.460428 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.459663 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:55:55.460428 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.459953 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-multus-daemon-config\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.460428 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.460065 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d1b84f62-7f09-42d0-8ed1-a2447efa3bf3-serviceca\") pod \"node-ca-9fpvq\" (UID: \"d1b84f62-7f09-42d0-8ed1-a2447efa3bf3\") " pod="openshift-image-registry/node-ca-9fpvq" Oct 30 06:55:55.460428 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.460172 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:55:55.460428 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.460238 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:55:55.461181 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.460446 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-ovnkube-config\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.461181 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.460823 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-ovnkube-script-lib\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.461181 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.460836 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-env-overrides\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.461893 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.461862 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0994513f-0dac-4106-8bfe-5a89cbc11f02-tmp\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.462250 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.462228 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-ovn-node-metrics-cert\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.465682 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.465659 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5225f97b-21b7-453c-8d19-ccfce5ee6d54-agent-certs\") pod \"konnectivity-agent-mjwrd\" (UID: \"5225f97b-21b7-453c-8d19-ccfce5ee6d54\") " pod="kube-system/konnectivity-agent-mjwrd" Oct 30 06:55:55.469746 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.469725 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-tuned\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.479466 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.479412 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlg4z\" (UniqueName: \"kubernetes.io/projected/d1b84f62-7f09-42d0-8ed1-a2447efa3bf3-kube-api-access-tlg4z\") pod \"node-ca-9fpvq\" (UID: \"d1b84f62-7f09-42d0-8ed1-a2447efa3bf3\") " pod="openshift-image-registry/node-ca-9fpvq" Oct 30 06:55:55.479974 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.479953 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bx54\" (UniqueName: \"kubernetes.io/projected/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-kube-api-access-2bx54\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:55:55.480171 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.480137 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbp46\" (UniqueName: \"kubernetes.io/projected/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-kube-api-access-xbp46\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.483156 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.483135 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v8q5\" (UniqueName: \"kubernetes.io/projected/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-kube-api-access-6v8q5\") pod \"network-metrics-daemon-g6vbv\" (UID: \"f1b14112-6c97-4bfa-8f2a-5988f59ad06a\") " pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:55:55.483505 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.483482 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l5w2\" (UniqueName: \"kubernetes.io/projected/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-kube-api-access-2l5w2\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.490913 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:55.490789 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 06:55:55.490913 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:55.490810 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 06:55:55.490913 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:55.490824 2574 projected.go:196] Error preparing data for projected volume kube-api-access-x7ndt for pod openshift-network-diagnostics/network-check-target-rjdmc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:55:55.490913 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:55.490891 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/16e1a28b-a156-4d4e-820f-8712616b4eb2-kube-api-access-x7ndt podName:16e1a28b-a156-4d4e-820f-8712616b4eb2 nodeName:}" failed. No retries permitted until 2026-10-30 06:55:55.990873618 +0000 UTC m=+3.120010130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-x7ndt" (UniqueName: "kubernetes.io/projected/16e1a28b-a156-4d4e-820f-8712616b4eb2-kube-api-access-x7ndt") pod "network-check-target-rjdmc" (UID: "16e1a28b-a156-4d4e-820f-8712616b4eb2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:55:55.492618 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.492558 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9jzs\" (UniqueName: \"kubernetes.io/projected/0994513f-0dac-4106-8bfe-5a89cbc11f02-kube-api-access-x9jzs\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.493089 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.493067 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6527\" (UniqueName: \"kubernetes.io/projected/4664d551-2cbf-42c5-a730-0d8c088f4f0f-kube-api-access-t6527\") pod \"node-resolver-qq6f5\" (UID: \"4664d551-2cbf-42c5-a730-0d8c088f4f0f\") " pod="openshift-dns/node-resolver-qq6f5" Oct 30 06:55:55.493428 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.493387 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj7gf\" (UniqueName: \"kubernetes.io/projected/bbda0b6e-80fb-4964-a0a9-23c69a84889e-kube-api-access-zj7gf\") pod \"iptables-alerter-zzvr7\" (UID: \"bbda0b6e-80fb-4964-a0a9-23c69a84889e\") " pod="openshift-network-operator/iptables-alerter-zzvr7" Oct 30 06:55:55.494324 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.494306 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbzjd\" (UniqueName: \"kubernetes.io/projected/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-kube-api-access-fbzjd\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:55:55.594853 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.594789 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:55:55.605348 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.605324 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:55:55.615249 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.615226 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qq6f5" Oct 30 06:55:55.616734 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.616717 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:55:55.620071 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.620055 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zzvr7" Oct 30 06:55:55.623978 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.623957 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:55:55.626007 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.625987 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mjwrd" Oct 30 06:55:55.626855 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:55.626831 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod044f4f8f_15bd_4f21_b6ca_2afa1f1171c5.slice/crio-5c7340e8e35fd6770010c21ea4f96f69e807d4a7ec693bc5ac261d0a2dc3fb36 WatchSource:0}: Error finding container 5c7340e8e35fd6770010c21ea4f96f69e807d4a7ec693bc5ac261d0a2dc3fb36: Status 404 returned error can't find the container with id 5c7340e8e35fd6770010c21ea4f96f69e807d4a7ec693bc5ac261d0a2dc3fb36 Oct 30 06:55:55.628131 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.628109 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9fpvq" Oct 30 06:55:55.632378 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:55.632355 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0994513f_0dac_4106_8bfe_5a89cbc11f02.slice/crio-a1f827b63147484ae4bd24060e01e7ff4ecbd77722dfd377517e569d4ee48e9d WatchSource:0}: Error finding container a1f827b63147484ae4bd24060e01e7ff4ecbd77722dfd377517e569d4ee48e9d: Status 404 returned error can't find the container with id a1f827b63147484ae4bd24060e01e7ff4ecbd77722dfd377517e569d4ee48e9d Oct 30 06:55:55.633712 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.633695 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q9jl6" Oct 30 06:55:55.636486 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:55.636460 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4664d551_2cbf_42c5_a730_0d8c088f4f0f.slice/crio-d28b9134ddbbda474c10a8fe5e7d167db5e1dc7f40556a8a82e354ec532eda0d WatchSource:0}: Error finding container d28b9134ddbbda474c10a8fe5e7d167db5e1dc7f40556a8a82e354ec532eda0d: Status 404 returned error can't find the container with id d28b9134ddbbda474c10a8fe5e7d167db5e1dc7f40556a8a82e354ec532eda0d Oct 30 06:55:55.638816 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:55.638786 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4aa6b79_3b44_4b45_834c_78ee0fb91e02.slice/crio-ca4c81fbca9c9457378519a84eb368f9d3124ad49a53a905b44508852582cec1 WatchSource:0}: Error finding container ca4c81fbca9c9457378519a84eb368f9d3124ad49a53a905b44508852582cec1: Status 404 returned error can't find the container with id ca4c81fbca9c9457378519a84eb368f9d3124ad49a53a905b44508852582cec1 Oct 30 06:55:55.646168 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:55.646143 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd315a45_3a31_44f2_9bf1_6d43df8e0c44.slice/crio-8a543b8d414ca434a8d735c2bdcc4e54a60233f3dc0b601715f9647d4168ffc5 WatchSource:0}: Error finding container 8a543b8d414ca434a8d735c2bdcc4e54a60233f3dc0b601715f9647d4168ffc5: Status 404 returned error can't find the container with id 8a543b8d414ca434a8d735c2bdcc4e54a60233f3dc0b601715f9647d4168ffc5 Oct 30 06:55:55.647803 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:55.647746 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbda0b6e_80fb_4964_a0a9_23c69a84889e.slice/crio-0eb6042ac52c60a5e6ea86f01e3f88c5940346322cb6aebc1d2c6cd7887c2634 WatchSource:0}: Error finding container 0eb6042ac52c60a5e6ea86f01e3f88c5940346322cb6aebc1d2c6cd7887c2634: Status 404 returned error can't find the container with id 0eb6042ac52c60a5e6ea86f01e3f88c5940346322cb6aebc1d2c6cd7887c2634 Oct 30 06:55:55.652687 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:55.652667 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5225f97b_21b7_453c_8d19_ccfce5ee6d54.slice/crio-fcc1d468d5e269d20f05df58a51e5fad95b30937d4608421d3ec434933576af1 WatchSource:0}: Error finding container fcc1d468d5e269d20f05df58a51e5fad95b30937d4608421d3ec434933576af1: Status 404 returned error can't find the container with id fcc1d468d5e269d20f05df58a51e5fad95b30937d4608421d3ec434933576af1 Oct 30 06:55:55.654064 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:55.654042 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1b84f62_7f09_42d0_8ed1_a2447efa3bf3.slice/crio-6d09172da1dab5fb0a61df183babe77f540ac253f67a3ed724be7ce751abd60b WatchSource:0}: Error finding container 6d09172da1dab5fb0a61df183babe77f540ac253f67a3ed724be7ce751abd60b: Status 404 returned error can't find the container with id 6d09172da1dab5fb0a61df183babe77f540ac253f67a3ed724be7ce751abd60b Oct 30 06:55:55.656853 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:55:55.656710 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3be0bd9d_9b78_40d8_bfcc_d0a591c108ce.slice/crio-1489b72942e5e7d0d5e7633a8c481841dd3fc31b4a356992b62ac27bbd2566c0 WatchSource:0}: Error finding container 1489b72942e5e7d0d5e7633a8c481841dd3fc31b4a356992b62ac27bbd2566c0: Status 404 returned error can't find the container with id 1489b72942e5e7d0d5e7633a8c481841dd3fc31b4a356992b62ac27bbd2566c0 Oct 30 06:55:55.960965 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.960932 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-metrics-certs\") pod \"network-metrics-daemon-g6vbv\" (UID: \"f1b14112-6c97-4bfa-8f2a-5988f59ad06a\") " pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:55:55.961045 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:55.960969 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-original-pull-secret\") pod \"global-pull-secret-syncer-ts77g\" (UID: \"96024683-dd4e-4d78-b5ad-6aa9ccc4bc05\") " pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:55:55.961107 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:55.961093 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 06:55:55.961160 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:55.961151 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-original-pull-secret podName:96024683-dd4e-4d78-b5ad-6aa9ccc4bc05 nodeName:}" failed. No retries permitted until 2026-10-30 06:55:56.961138489 +0000 UTC m=+4.090274982 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-original-pull-secret") pod "global-pull-secret-syncer-ts77g" (UID: "96024683-dd4e-4d78-b5ad-6aa9ccc4bc05") : object "kube-system"/"original-pull-secret" not registered Oct 30 06:55:55.961209 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:55.961093 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:55:55.961209 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:55.961195 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-metrics-certs podName:f1b14112-6c97-4bfa-8f2a-5988f59ad06a nodeName:}" failed. No retries permitted until 2026-10-30 06:55:56.96118559 +0000 UTC m=+4.090322085 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-metrics-certs") pod "network-metrics-daemon-g6vbv" (UID: "f1b14112-6c97-4bfa-8f2a-5988f59ad06a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:55:56.061915 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:56.061542 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7ndt\" (UniqueName: \"kubernetes.io/projected/16e1a28b-a156-4d4e-820f-8712616b4eb2-kube-api-access-x7ndt\") pod \"network-check-target-rjdmc\" (UID: \"16e1a28b-a156-4d4e-820f-8712616b4eb2\") " pod="openshift-network-diagnostics/network-check-target-rjdmc" Oct 30 06:55:56.061915 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:56.061780 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 06:55:56.061915 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:56.061803 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 06:55:56.061915 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:56.061816 2574 projected.go:196] Error preparing data for projected volume kube-api-access-x7ndt for pod openshift-network-diagnostics/network-check-target-rjdmc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:55:56.061915 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:56.061875 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/16e1a28b-a156-4d4e-820f-8712616b4eb2-kube-api-access-x7ndt podName:16e1a28b-a156-4d4e-820f-8712616b4eb2 nodeName:}" failed. No retries permitted until 2026-10-30 06:55:57.061856382 +0000 UTC m=+4.190992886 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-x7ndt" (UniqueName: "kubernetes.io/projected/16e1a28b-a156-4d4e-820f-8712616b4eb2-kube-api-access-x7ndt") pod "network-check-target-rjdmc" (UID: "16e1a28b-a156-4d4e-820f-8712616b4eb2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:55:56.310028 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:56.309958 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 06:50:55 +0000 UTC" deadline="2028-04-08 06:40:58.742244114 +0000 UTC" Oct 30 06:55:56.310028 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:56.310023 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12623h45m2.432224898s" Oct 30 06:55:56.393187 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:56.392692 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" event={"ID":"5b1dc20b8a7bd35e96a3f207cff8defc","Type":"ContainerStarted","Data":"32ce3a26e85848f3e37211dff87c5e578eefe1cb83b3a9c9875180d85156fed0"} Oct 30 06:55:56.396610 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:56.396560 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9fpvq" event={"ID":"d1b84f62-7f09-42d0-8ed1-a2447efa3bf3","Type":"ContainerStarted","Data":"6d09172da1dab5fb0a61df183babe77f540ac253f67a3ed724be7ce751abd60b"} Oct 30 06:55:56.414330 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:56.414282 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" event={"ID":"bd315a45-3a31-44f2-9bf1-6d43df8e0c44","Type":"ContainerStarted","Data":"8a543b8d414ca434a8d735c2bdcc4e54a60233f3dc0b601715f9647d4168ffc5"} Oct 30 06:55:56.420940 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:56.420897 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" event={"ID":"a4aa6b79-3b44-4b45-834c-78ee0fb91e02","Type":"ContainerStarted","Data":"ca4c81fbca9c9457378519a84eb368f9d3124ad49a53a905b44508852582cec1"} Oct 30 06:55:56.424726 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:56.424692 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qq6f5" event={"ID":"4664d551-2cbf-42c5-a730-0d8c088f4f0f","Type":"ContainerStarted","Data":"d28b9134ddbbda474c10a8fe5e7d167db5e1dc7f40556a8a82e354ec532eda0d"} Oct 30 06:55:56.427968 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:56.427932 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" event={"ID":"0994513f-0dac-4106-8bfe-5a89cbc11f02","Type":"ContainerStarted","Data":"a1f827b63147484ae4bd24060e01e7ff4ecbd77722dfd377517e569d4ee48e9d"} Oct 30 06:55:56.429816 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:56.429782 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4rnr" event={"ID":"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5","Type":"ContainerStarted","Data":"5c7340e8e35fd6770010c21ea4f96f69e807d4a7ec693bc5ac261d0a2dc3fb36"} Oct 30 06:55:56.439456 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:56.438555 2574 generic.go:356] "Generic (PLEG): container finished" podID="6ad70715366d9db6ff14ef77cdc50313" containerID="851007e35aaa6fe3639c9c48fb3962da44b84fce2bbcfcbfda033d3759d2e159" exitCode=0 Oct 30 06:55:56.439456 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:56.439399 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" event={"ID":"6ad70715366d9db6ff14ef77cdc50313","Type":"ContainerDied","Data":"851007e35aaa6fe3639c9c48fb3962da44b84fce2bbcfcbfda033d3759d2e159"} Oct 30 06:55:56.448385 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:56.448350 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q9jl6" event={"ID":"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce","Type":"ContainerStarted","Data":"1489b72942e5e7d0d5e7633a8c481841dd3fc31b4a356992b62ac27bbd2566c0"} Oct 30 06:55:56.480518 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:56.480443 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" podStartSLOduration=1.480422378 podStartE2EDuration="1.480422378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:55:56.432884612 +0000 UTC m=+3.562021129" watchObservedRunningTime="2026-10-30 06:55:56.480422378 +0000 UTC m=+3.609558895" Oct 30 06:55:56.500060 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:56.499999 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mjwrd" event={"ID":"5225f97b-21b7-453c-8d19-ccfce5ee6d54","Type":"ContainerStarted","Data":"fcc1d468d5e269d20f05df58a51e5fad95b30937d4608421d3ec434933576af1"} Oct 30 06:55:56.517266 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:56.517229 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zzvr7" event={"ID":"bbda0b6e-80fb-4964-a0a9-23c69a84889e","Type":"ContainerStarted","Data":"0eb6042ac52c60a5e6ea86f01e3f88c5940346322cb6aebc1d2c6cd7887c2634"} Oct 30 06:55:56.971351 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:56.971038 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-metrics-certs\") pod \"network-metrics-daemon-g6vbv\" (UID: \"f1b14112-6c97-4bfa-8f2a-5988f59ad06a\") " pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:55:56.971351 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:56.971303 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-original-pull-secret\") pod \"global-pull-secret-syncer-ts77g\" (UID: \"96024683-dd4e-4d78-b5ad-6aa9ccc4bc05\") " pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:55:56.971564 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:56.971472 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 06:55:56.971564 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:56.971534 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-original-pull-secret podName:96024683-dd4e-4d78-b5ad-6aa9ccc4bc05 nodeName:}" failed. No retries permitted until 2026-10-30 06:55:58.971516988 +0000 UTC m=+6.100653499 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-original-pull-secret") pod "global-pull-secret-syncer-ts77g" (UID: "96024683-dd4e-4d78-b5ad-6aa9ccc4bc05") : object "kube-system"/"original-pull-secret" not registered Oct 30 06:55:56.971964 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:56.971946 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:55:56.972032 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:56.971999 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-metrics-certs podName:f1b14112-6c97-4bfa-8f2a-5988f59ad06a nodeName:}" failed. No retries permitted until 2026-10-30 06:55:58.971983106 +0000 UTC m=+6.101119603 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-metrics-certs") pod "network-metrics-daemon-g6vbv" (UID: "f1b14112-6c97-4bfa-8f2a-5988f59ad06a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:55:57.072566 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:57.072526 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7ndt\" (UniqueName: \"kubernetes.io/projected/16e1a28b-a156-4d4e-820f-8712616b4eb2-kube-api-access-x7ndt\") pod \"network-check-target-rjdmc\" (UID: \"16e1a28b-a156-4d4e-820f-8712616b4eb2\") " pod="openshift-network-diagnostics/network-check-target-rjdmc" Oct 30 06:55:57.072791 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:57.072766 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 06:55:57.072791 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:57.072789 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 06:55:57.072898 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:57.072803 2574 projected.go:196] Error preparing data for projected volume kube-api-access-x7ndt for pod openshift-network-diagnostics/network-check-target-rjdmc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:55:57.072898 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:57.072866 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/16e1a28b-a156-4d4e-820f-8712616b4eb2-kube-api-access-x7ndt podName:16e1a28b-a156-4d4e-820f-8712616b4eb2 nodeName:}" failed. No retries permitted until 2026-10-30 06:55:59.072848777 +0000 UTC m=+6.201985274 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-x7ndt" (UniqueName: "kubernetes.io/projected/16e1a28b-a156-4d4e-820f-8712616b4eb2-kube-api-access-x7ndt") pod "network-check-target-rjdmc" (UID: "16e1a28b-a156-4d4e-820f-8712616b4eb2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:55:57.311128 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:57.311067 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 06:50:55 +0000 UTC" deadline="2028-07-12 21:51:26.053140612 +0000 UTC" Oct 30 06:55:57.311128 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:57.311099 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14918h55m28.742044922s" Oct 30 06:55:57.370218 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:57.369720 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:55:57.370218 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:57.369783 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjdmc" Oct 30 06:55:57.370218 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:57.369793 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:55:57.370218 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:57.369885 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjdmc" podUID="16e1a28b-a156-4d4e-820f-8712616b4eb2" Oct 30 06:55:57.370218 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:57.369983 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts77g" podUID="96024683-dd4e-4d78-b5ad-6aa9ccc4bc05" Oct 30 06:55:57.370218 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:57.370066 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6vbv" podUID="f1b14112-6c97-4bfa-8f2a-5988f59ad06a" Oct 30 06:55:57.554798 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:57.554712 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" event={"ID":"6ad70715366d9db6ff14ef77cdc50313","Type":"ContainerStarted","Data":"d37ee28df8c2343082e62d483b7145a8aad5f31d2e4542c364beaf8940b2d106"} Oct 30 06:55:58.571405 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:58.569901 2574 generic.go:356] "Generic (PLEG): container finished" podID="044f4f8f-15bd-4f21-b6ca-2afa1f1171c5" containerID="b9d8a19cd0437c48fd36c921903a8e0b5fc23aef9b9568d9c091831a36f35f17" exitCode=0 Oct 30 06:55:58.571405 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:58.571119 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4rnr" event={"ID":"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5","Type":"ContainerDied","Data":"b9d8a19cd0437c48fd36c921903a8e0b5fc23aef9b9568d9c091831a36f35f17"} Oct 30 06:55:58.643114 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:58.643036 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" podStartSLOduration=3.643013968 podStartE2EDuration="3.643013968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:55:57.595595119 +0000 UTC m=+4.724731644" watchObservedRunningTime="2026-10-30 06:55:58.643013968 +0000 UTC m=+5.772150486" Oct 30 06:55:58.990025 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:58.989935 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-metrics-certs\") pod \"network-metrics-daemon-g6vbv\" (UID: \"f1b14112-6c97-4bfa-8f2a-5988f59ad06a\") " pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:55:58.990025 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:58.990001 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-original-pull-secret\") pod \"global-pull-secret-syncer-ts77g\" (UID: \"96024683-dd4e-4d78-b5ad-6aa9ccc4bc05\") " pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:55:58.990218 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:58.990174 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 06:55:58.990250 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:58.990240 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-original-pull-secret podName:96024683-dd4e-4d78-b5ad-6aa9ccc4bc05 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:02.990222472 +0000 UTC m=+10.119358966 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-original-pull-secret") pod "global-pull-secret-syncer-ts77g" (UID: "96024683-dd4e-4d78-b5ad-6aa9ccc4bc05") : object "kube-system"/"original-pull-secret" not registered Oct 30 06:55:58.990316 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:58.990303 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:55:58.990358 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:58.990343 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-metrics-certs podName:f1b14112-6c97-4bfa-8f2a-5988f59ad06a nodeName:}" failed. No retries permitted until 2026-10-30 06:56:02.990331875 +0000 UTC m=+10.119468372 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-metrics-certs") pod "network-metrics-daemon-g6vbv" (UID: "f1b14112-6c97-4bfa-8f2a-5988f59ad06a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:55:59.091413 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:59.091348 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7ndt\" (UniqueName: \"kubernetes.io/projected/16e1a28b-a156-4d4e-820f-8712616b4eb2-kube-api-access-x7ndt\") pod \"network-check-target-rjdmc\" (UID: \"16e1a28b-a156-4d4e-820f-8712616b4eb2\") " pod="openshift-network-diagnostics/network-check-target-rjdmc" Oct 30 06:55:59.091549 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:59.091528 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 06:55:59.091549 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:59.091548 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 06:55:59.091643 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:59.091561 2574 projected.go:196] Error preparing data for projected volume kube-api-access-x7ndt for pod openshift-network-diagnostics/network-check-target-rjdmc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:55:59.091643 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:59.091633 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/16e1a28b-a156-4d4e-820f-8712616b4eb2-kube-api-access-x7ndt podName:16e1a28b-a156-4d4e-820f-8712616b4eb2 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:03.091615855 +0000 UTC m=+10.220752362 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-x7ndt" (UniqueName: "kubernetes.io/projected/16e1a28b-a156-4d4e-820f-8712616b4eb2-kube-api-access-x7ndt") pod "network-check-target-rjdmc" (UID: "16e1a28b-a156-4d4e-820f-8712616b4eb2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:55:59.372250 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:59.372209 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjdmc" Oct 30 06:55:59.372417 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:59.372346 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjdmc" podUID="16e1a28b-a156-4d4e-820f-8712616b4eb2" Oct 30 06:55:59.372479 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:59.372426 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:55:59.372561 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:59.372514 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6vbv" podUID="f1b14112-6c97-4bfa-8f2a-5988f59ad06a" Oct 30 06:55:59.372648 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:55:59.372588 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:55:59.372700 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:55:59.372674 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts77g" podUID="96024683-dd4e-4d78-b5ad-6aa9ccc4bc05" Oct 30 06:56:00.577885 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:00.577621 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mjwrd" event={"ID":"5225f97b-21b7-453c-8d19-ccfce5ee6d54","Type":"ContainerStarted","Data":"ac74a4c8948f7556c969938d7d94ebc986515aaf258d2a331d21863ab230c169"} Oct 30 06:56:01.370736 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:01.370661 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:56:01.370871 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:01.370783 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6vbv" podUID="f1b14112-6c97-4bfa-8f2a-5988f59ad06a" Oct 30 06:56:01.371090 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:01.371071 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:56:01.371179 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:01.371142 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts77g" podUID="96024683-dd4e-4d78-b5ad-6aa9ccc4bc05" Oct 30 06:56:01.371281 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:01.371268 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjdmc" Oct 30 06:56:01.371374 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:01.371356 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjdmc" podUID="16e1a28b-a156-4d4e-820f-8712616b4eb2" Oct 30 06:56:01.585749 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:01.585711 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" event={"ID":"a4aa6b79-3b44-4b45-834c-78ee0fb91e02","Type":"ContainerStarted","Data":"88d7b3cbd8ade2e8881336445fc00442c84972beaf57760f26c863b822420b63"} Oct 30 06:56:01.588858 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:01.588801 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9fpvq" event={"ID":"d1b84f62-7f09-42d0-8ed1-a2447efa3bf3","Type":"ContainerStarted","Data":"db17291ea8ef6ba3d2e0b3d9aa17ecf406a144fc0b743283693d35fa223723d2"} Oct 30 06:56:01.629234 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:01.628730 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mjwrd" podStartSLOduration=4.87767902 podStartE2EDuration="8.628714369s" totalImagesPullingTime="3.751035349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:55:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:00.617548816 +0000 UTC m=+7.746685331" watchObservedRunningTime="2026-10-30 06:56:01.628714369 +0000 UTC m=+8.757850883" Oct 30 06:56:02.696254 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:02.696209 2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 06:56:03.029079 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:03.029044 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-metrics-certs\") pod \"network-metrics-daemon-g6vbv\" (UID: \"f1b14112-6c97-4bfa-8f2a-5988f59ad06a\") " pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:56:03.029251 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:03.029101 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-original-pull-secret\") pod \"global-pull-secret-syncer-ts77g\" (UID: \"96024683-dd4e-4d78-b5ad-6aa9ccc4bc05\") " pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:56:03.029251 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:03.029228 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 06:56:03.029351 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:03.029229 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:56:03.029351 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:03.029295 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-original-pull-secret podName:96024683-dd4e-4d78-b5ad-6aa9ccc4bc05 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:11.029278622 +0000 UTC m=+18.158415121 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-original-pull-secret") pod "global-pull-secret-syncer-ts77g" (UID: "96024683-dd4e-4d78-b5ad-6aa9ccc4bc05") : object "kube-system"/"original-pull-secret" not registered Oct 30 06:56:03.029351 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:03.029343 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-metrics-certs podName:f1b14112-6c97-4bfa-8f2a-5988f59ad06a nodeName:}" failed. No retries permitted until 2026-10-30 06:56:11.029326968 +0000 UTC m=+18.158463475 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-metrics-certs") pod "network-metrics-daemon-g6vbv" (UID: "f1b14112-6c97-4bfa-8f2a-5988f59ad06a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:56:03.130455 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:03.130404 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7ndt\" (UniqueName: \"kubernetes.io/projected/16e1a28b-a156-4d4e-820f-8712616b4eb2-kube-api-access-x7ndt\") pod \"network-check-target-rjdmc\" (UID: \"16e1a28b-a156-4d4e-820f-8712616b4eb2\") " pod="openshift-network-diagnostics/network-check-target-rjdmc" Oct 30 06:56:03.130703 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:03.130590 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 06:56:03.130703 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:03.130609 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 06:56:03.130703 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:03.130620 2574 projected.go:196] Error preparing data for projected volume kube-api-access-x7ndt for pod openshift-network-diagnostics/network-check-target-rjdmc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:56:03.130703 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:03.130683 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/16e1a28b-a156-4d4e-820f-8712616b4eb2-kube-api-access-x7ndt podName:16e1a28b-a156-4d4e-820f-8712616b4eb2 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:11.130667112 +0000 UTC m=+18.259803619 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-x7ndt" (UniqueName: "kubernetes.io/projected/16e1a28b-a156-4d4e-820f-8712616b4eb2-kube-api-access-x7ndt") pod "network-check-target-rjdmc" (UID: "16e1a28b-a156-4d4e-820f-8712616b4eb2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:56:03.306040 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:03.305944 2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T06:56:02.696223763Z","UUID":"071dec25-fa4e-4aa5-8ea3-9a360f4f327c","Handler":null,"Name":"","Endpoint":""} Oct 30 06:56:03.308399 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:03.307877 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 06:56:03.308399 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:03.307902 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 06:56:03.371726 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:03.371694 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:56:03.371833 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:03.371737 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjdmc" Oct 30 06:56:03.371833 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:03.371816 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts77g" podUID="96024683-dd4e-4d78-b5ad-6aa9ccc4bc05" Oct 30 06:56:03.371927 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:03.371694 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:56:03.371991 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:03.371932 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6vbv" podUID="f1b14112-6c97-4bfa-8f2a-5988f59ad06a" Oct 30 06:56:03.372045 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:03.372001 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjdmc" podUID="16e1a28b-a156-4d4e-820f-8712616b4eb2" Oct 30 06:56:03.597965 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:03.597926 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" event={"ID":"a4aa6b79-3b44-4b45-834c-78ee0fb91e02","Type":"ContainerStarted","Data":"67e89e0d254d186dff334f9cef0dcd040d7b47fa45f63625aaa0ec18373d5ef1"} Oct 30 06:56:03.600269 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:03.600235 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qq6f5" event={"ID":"4664d551-2cbf-42c5-a730-0d8c088f4f0f","Type":"ContainerStarted","Data":"d80d451b5cb0711713e4578a40c5e2a945ac2af4b065ad23201c5f62a42e519d"} Oct 30 06:56:03.603898 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:03.603860 2574 generic.go:356] "Generic (PLEG): container finished" podID="044f4f8f-15bd-4f21-b6ca-2afa1f1171c5" containerID="ef9501af39d44bc8804d23d6b5cea138908726a5ec12b2fc25a648e1cb1b748d" exitCode=0 Oct 30 06:56:03.604028 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:03.603965 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4rnr" event={"ID":"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5","Type":"ContainerDied","Data":"ef9501af39d44bc8804d23d6b5cea138908726a5ec12b2fc25a648e1cb1b748d"} Oct 30 06:56:03.606055 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:03.605922 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q9jl6" event={"ID":"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce","Type":"ContainerStarted","Data":"10a5da8b5f69e5a90cca3e4633f6552817ae3a1e6de59b94510b7d7dbe8ef20a"} Oct 30 06:56:03.639783 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:03.639730 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-qq6f5" podStartSLOduration=2.924651168 podStartE2EDuration="10.639717244s" totalImagesPullingTime="7.715066076s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:55:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:03.639580503 +0000 UTC m=+10.768717009" watchObservedRunningTime="2026-10-30 06:56:03.639717244 +0000 UTC m=+10.768853759" Oct 30 06:56:03.639953 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:03.639931 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9fpvq" podStartSLOduration=5.913951403 podStartE2EDuration="10.639924098s" totalImagesPullingTime="4.725972695s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:55:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:01.629160917 +0000 UTC m=+8.758297433" watchObservedRunningTime="2026-10-30 06:56:03.639924098 +0000 UTC m=+10.769060645" Oct 30 06:56:03.727008 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:03.726947 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-q9jl6" podStartSLOduration=3.142986852 podStartE2EDuration="10.726926362s" totalImagesPullingTime="7.58393951s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:55:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:03.681101227 +0000 UTC m=+10.810237744" watchObservedRunningTime="2026-10-30 06:56:03.726926362 +0000 UTC m=+10.856062878" Oct 30 06:56:04.377951 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:04.377709 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mjwrd" Oct 30 06:56:04.379075 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:04.379050 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mjwrd" Oct 30 06:56:04.611554 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:04.611516 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" event={"ID":"a4aa6b79-3b44-4b45-834c-78ee0fb91e02","Type":"ContainerStarted","Data":"0c77a1edd058113b5f2c7a1950fa5d5c1e34c2bfa07078549f8ae8001ec563a3"} Oct 30 06:56:04.614094 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:04.614058 2574 generic.go:356] "Generic (PLEG): container finished" podID="044f4f8f-15bd-4f21-b6ca-2afa1f1171c5" containerID="554723614f7fea2c063968e51c10eb1bff40141e57654959c121ebcb253a849f" exitCode=0 Oct 30 06:56:04.614209 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:04.614170 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4rnr" event={"ID":"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5","Type":"ContainerDied","Data":"554723614f7fea2c063968e51c10eb1bff40141e57654959c121ebcb253a849f"} Oct 30 06:56:04.657060 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:04.656978 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" podStartSLOduration=3.9503390019999998 podStartE2EDuration="11.656956929s" totalImagesPullingTime="7.706617927s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:55:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:04.654661049 +0000 UTC m=+11.783797560" watchObservedRunningTime="2026-10-30 06:56:04.656956929 +0000 UTC m=+11.786093445" Oct 30 06:56:05.370100 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:05.370037 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:56:05.370100 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:05.370079 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjdmc" Oct 30 06:56:05.370100 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:05.370090 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:56:05.370490 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:05.370228 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6vbv" podUID="f1b14112-6c97-4bfa-8f2a-5988f59ad06a" Oct 30 06:56:05.370490 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:05.370317 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjdmc" podUID="16e1a28b-a156-4d4e-820f-8712616b4eb2" Oct 30 06:56:05.370490 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:05.370389 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts77g" podUID="96024683-dd4e-4d78-b5ad-6aa9ccc4bc05" Oct 30 06:56:05.618112 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:05.618025 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zzvr7" event={"ID":"bbda0b6e-80fb-4964-a0a9-23c69a84889e","Type":"ContainerStarted","Data":"004c9e228a843ff4aac386d027d8930fc0c6bebd3afce5bad4c20912b1538e79"} Oct 30 06:56:05.619989 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:05.619959 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" event={"ID":"0994513f-0dac-4106-8bfe-5a89cbc11f02","Type":"ContainerStarted","Data":"b9a6ea31f2dc616ba42406fd8751eaa83b43c60ade1fd77cd3ea6db915b01f41"} Oct 30 06:56:05.623071 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:05.623041 2574 generic.go:356] "Generic (PLEG): container finished" podID="044f4f8f-15bd-4f21-b6ca-2afa1f1171c5" containerID="e4886fb1f9c4052deeb996ecb4baca4a76b47b10853941a7c51db8ce8fbc7be3" exitCode=0 Oct 30 06:56:05.623198 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:05.623130 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4rnr" event={"ID":"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5","Type":"ContainerDied","Data":"e4886fb1f9c4052deeb996ecb4baca4a76b47b10853941a7c51db8ce8fbc7be3"} Oct 30 06:56:05.657686 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:05.657639 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zzvr7" podStartSLOduration=4.763326466 podStartE2EDuration="12.657621915s" totalImagesPullingTime="7.894295449s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:55:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:05.657253527 +0000 UTC m=+12.786390044" watchObservedRunningTime="2026-10-30 06:56:05.657621915 +0000 UTC m=+12.786758432" Oct 30 06:56:05.780775 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:05.780728 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" podStartSLOduration=3.05297255 podStartE2EDuration="12.780715239s" totalImagesPullingTime="9.727742689s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:55:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:05.780694239 +0000 UTC m=+12.909830754" watchObservedRunningTime="2026-10-30 06:56:05.780715239 +0000 UTC m=+12.909851754" Oct 30 06:56:07.370364 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:07.370329 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:56:07.371195 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:07.370390 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjdmc" Oct 30 06:56:07.371195 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:07.370403 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:56:07.371195 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:07.370495 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts77g" podUID="96024683-dd4e-4d78-b5ad-6aa9ccc4bc05" Oct 30 06:56:07.371195 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:07.370892 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjdmc" podUID="16e1a28b-a156-4d4e-820f-8712616b4eb2" Oct 30 06:56:07.371195 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:07.370981 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6vbv" podUID="f1b14112-6c97-4bfa-8f2a-5988f59ad06a" Oct 30 06:56:07.631537 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:07.631508 2574 generic.go:356] "Generic (PLEG): container finished" podID="044f4f8f-15bd-4f21-b6ca-2afa1f1171c5" containerID="8a9e3a57a54719c70ece10510e62315ce8b502c7af510bb0fd91df90ca25d6e0" exitCode=0 Oct 30 06:56:07.631711 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:07.631543 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4rnr" event={"ID":"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5","Type":"ContainerDied","Data":"8a9e3a57a54719c70ece10510e62315ce8b502c7af510bb0fd91df90ca25d6e0"} Oct 30 06:56:08.638154 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:08.638118 2574 generic.go:356] "Generic (PLEG): container finished" podID="044f4f8f-15bd-4f21-b6ca-2afa1f1171c5" containerID="72ca7c79831f83085c6d6248110e1223874dcc00d9aac07ac2ba17a3082839e7" exitCode=0 Oct 30 06:56:08.638776 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:08.638186 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4rnr" event={"ID":"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5","Type":"ContainerDied","Data":"72ca7c79831f83085c6d6248110e1223874dcc00d9aac07ac2ba17a3082839e7"} Oct 30 06:56:09.370065 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:09.369874 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:56:09.370243 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:09.369877 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjdmc" Oct 30 06:56:09.370243 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:09.370145 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts77g" podUID="96024683-dd4e-4d78-b5ad-6aa9ccc4bc05" Oct 30 06:56:09.370243 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:09.369877 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:56:09.370243 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:09.370215 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjdmc" podUID="16e1a28b-a156-4d4e-820f-8712616b4eb2" Oct 30 06:56:09.370410 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:09.370283 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6vbv" podUID="f1b14112-6c97-4bfa-8f2a-5988f59ad06a" Oct 30 06:56:09.643236 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:09.643157 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4rnr" event={"ID":"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5","Type":"ContainerStarted","Data":"777e5f248c5f81cdbbcd7cb1b40bb6223d2b836b55913bff58dc5d4f035909a4"} Oct 30 06:56:09.717775 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:09.717716 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-t4rnr" podStartSLOduration=7.225552316 podStartE2EDuration="16.717696626s" totalImagesPullingTime="9.49214431s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:55:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:09.701984738 +0000 UTC m=+16.831121252" watchObservedRunningTime="2026-10-30 06:56:09.717696626 +0000 UTC m=+16.846833135" Oct 30 06:56:10.647892 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:10.647733 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 06:56:10.648314 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:10.648214 2574 generic.go:356] "Generic (PLEG): container finished" podID="bd315a45-3a31-44f2-9bf1-6d43df8e0c44" containerID="52d10454fd32559fd11d86f286c2497f3c9d8bf4db566724c2df2a9174fd166b" exitCode=1 Oct 30 06:56:10.648314 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:10.648284 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" event={"ID":"bd315a45-3a31-44f2-9bf1-6d43df8e0c44","Type":"ContainerStarted","Data":"b88e753ff126385679dc21240f41ac93899ba9dfa369787d6f90637b63262e2c"} Oct 30 06:56:10.648440 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:10.648367 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" event={"ID":"bd315a45-3a31-44f2-9bf1-6d43df8e0c44","Type":"ContainerStarted","Data":"f0ac4c6ff61c8bdbe073c18110776429bc6f98b328e58b8378bd54e6bc1d408b"} Oct 30 06:56:10.648440 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:10.648397 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" event={"ID":"bd315a45-3a31-44f2-9bf1-6d43df8e0c44","Type":"ContainerStarted","Data":"7f3f8acdc0cd0ad12d80261ae1e594b20c074cfe87c90252323f27faf0332717"} Oct 30 06:56:10.648440 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:10.648425 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" event={"ID":"bd315a45-3a31-44f2-9bf1-6d43df8e0c44","Type":"ContainerStarted","Data":"d6eee910e809c6cef124af7826b0c5728ee6cba10e108b635ae16d81434e8b71"} Oct 30 06:56:10.648599 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:10.648453 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" event={"ID":"bd315a45-3a31-44f2-9bf1-6d43df8e0c44","Type":"ContainerDied","Data":"52d10454fd32559fd11d86f286c2497f3c9d8bf4db566724c2df2a9174fd166b"} Oct 30 06:56:10.648599 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:10.648482 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" event={"ID":"bd315a45-3a31-44f2-9bf1-6d43df8e0c44","Type":"ContainerStarted","Data":"da757fde77683396236023daf8cec692b61850b3eeed569778cf1846eab7a58d"} Oct 30 06:56:11.093505 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:11.093475 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-metrics-certs\") pod \"network-metrics-daemon-g6vbv\" (UID: \"f1b14112-6c97-4bfa-8f2a-5988f59ad06a\") " pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:56:11.093677 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:11.093511 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-original-pull-secret\") pod \"global-pull-secret-syncer-ts77g\" (UID: \"96024683-dd4e-4d78-b5ad-6aa9ccc4bc05\") " pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:56:11.093677 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:11.093629 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 06:56:11.093677 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:11.093641 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:56:11.093677 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:11.093676 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-original-pull-secret podName:96024683-dd4e-4d78-b5ad-6aa9ccc4bc05 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:27.093664338 +0000 UTC m=+34.222800832 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-original-pull-secret") pod "global-pull-secret-syncer-ts77g" (UID: "96024683-dd4e-4d78-b5ad-6aa9ccc4bc05") : object "kube-system"/"original-pull-secret" not registered Oct 30 06:56:11.093835 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:11.093697 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-metrics-certs podName:f1b14112-6c97-4bfa-8f2a-5988f59ad06a nodeName:}" failed. No retries permitted until 2026-10-30 06:56:27.093681762 +0000 UTC m=+34.222818274 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-metrics-certs") pod "network-metrics-daemon-g6vbv" (UID: "f1b14112-6c97-4bfa-8f2a-5988f59ad06a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:56:11.194618 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:11.194589 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7ndt\" (UniqueName: \"kubernetes.io/projected/16e1a28b-a156-4d4e-820f-8712616b4eb2-kube-api-access-x7ndt\") pod \"network-check-target-rjdmc\" (UID: \"16e1a28b-a156-4d4e-820f-8712616b4eb2\") " pod="openshift-network-diagnostics/network-check-target-rjdmc" Oct 30 06:56:11.194751 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:11.194710 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 06:56:11.194751 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:11.194728 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 06:56:11.194751 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:11.194739 2574 projected.go:196] Error preparing data for projected volume kube-api-access-x7ndt for pod openshift-network-diagnostics/network-check-target-rjdmc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:56:11.194888 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:11.194793 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/16e1a28b-a156-4d4e-820f-8712616b4eb2-kube-api-access-x7ndt podName:16e1a28b-a156-4d4e-820f-8712616b4eb2 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:27.194775524 +0000 UTC m=+34.323912017 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-x7ndt" (UniqueName: "kubernetes.io/projected/16e1a28b-a156-4d4e-820f-8712616b4eb2-kube-api-access-x7ndt") pod "network-check-target-rjdmc" (UID: "16e1a28b-a156-4d4e-820f-8712616b4eb2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:56:11.369933 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:11.369850 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:56:11.369933 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:11.369885 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:56:11.370148 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:11.369934 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts77g" podUID="96024683-dd4e-4d78-b5ad-6aa9ccc4bc05" Oct 30 06:56:11.370148 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:11.369977 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6vbv" podUID="f1b14112-6c97-4bfa-8f2a-5988f59ad06a" Oct 30 06:56:11.370148 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:11.370028 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjdmc" Oct 30 06:56:11.370148 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:11.370079 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjdmc" podUID="16e1a28b-a156-4d4e-820f-8712616b4eb2" Oct 30 06:56:12.655275 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:12.655245 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 06:56:12.655814 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:12.655663 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" event={"ID":"bd315a45-3a31-44f2-9bf1-6d43df8e0c44","Type":"ContainerStarted","Data":"8635685227bd19a3cc875d58e4f8d8da35a7e19dff4fafb5426d413a5e94e81a"} Oct 30 06:56:13.370644 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:13.370617 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjdmc" Oct 30 06:56:13.370787 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:13.370694 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjdmc" podUID="16e1a28b-a156-4d4e-820f-8712616b4eb2" Oct 30 06:56:13.370787 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:13.370714 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:56:13.370787 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:13.370740 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:56:13.370890 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:13.370827 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts77g" podUID="96024683-dd4e-4d78-b5ad-6aa9ccc4bc05" Oct 30 06:56:13.370921 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:13.370908 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6vbv" podUID="f1b14112-6c97-4bfa-8f2a-5988f59ad06a" Oct 30 06:56:14.662677 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:14.662651 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 06:56:14.663433 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:14.662998 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" event={"ID":"bd315a45-3a31-44f2-9bf1-6d43df8e0c44","Type":"ContainerStarted","Data":"d5d486d51f1f561bf80e49f3c8280f9f98f289450b7e71bff8e5e477bba9fbbb"} Oct 30 06:56:14.663433 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:14.663256 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:14.663433 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:14.663339 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:14.663433 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:14.663360 2574 scope.go:122] "RemoveContainer" containerID="52d10454fd32559fd11d86f286c2497f3c9d8bf4db566724c2df2a9174fd166b" Oct 30 06:56:14.663433 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:14.663390 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:14.679965 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:14.679947 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:14.683039 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:14.683018 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:15.369767 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:15.369513 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:56:15.369987 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:15.369513 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjdmc" Oct 30 06:56:15.369987 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:15.369896 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts77g" podUID="96024683-dd4e-4d78-b5ad-6aa9ccc4bc05" Oct 30 06:56:15.369987 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:15.369539 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:56:15.369987 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:15.369920 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjdmc" podUID="16e1a28b-a156-4d4e-820f-8712616b4eb2" Oct 30 06:56:15.370120 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:15.370007 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6vbv" podUID="f1b14112-6c97-4bfa-8f2a-5988f59ad06a" Oct 30 06:56:15.668761 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:15.668667 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 06:56:15.669164 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:15.669010 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" event={"ID":"bd315a45-3a31-44f2-9bf1-6d43df8e0c44","Type":"ContainerStarted","Data":"e8ca9bf679e5f4de8f0c0d382a93cd48f134abec832d33683e8fea73fc621ae6"} Oct 30 06:56:15.736838 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:15.736782 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" podStartSLOduration=8.646586334 podStartE2EDuration="22.736765408s" totalImagesPullingTime="14.090179074s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:55:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:15.736005682 +0000 UTC m=+22.865142200" watchObservedRunningTime="2026-10-30 06:56:15.736765408 +0000 UTC m=+22.865901923" Oct 30 06:56:16.371742 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:16.371706 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ts77g"] Oct 30 06:56:16.371933 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:16.371813 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:56:16.371933 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:16.371895 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts77g" podUID="96024683-dd4e-4d78-b5ad-6aa9ccc4bc05" Oct 30 06:56:16.374457 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:16.374427 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rjdmc"] Oct 30 06:56:16.374603 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:16.374537 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjdmc" Oct 30 06:56:16.374676 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:16.374630 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjdmc" podUID="16e1a28b-a156-4d4e-820f-8712616b4eb2" Oct 30 06:56:16.378649 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:16.378624 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g6vbv"] Oct 30 06:56:16.378784 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:16.378713 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:56:16.378850 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:16.378799 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6vbv" podUID="f1b14112-6c97-4bfa-8f2a-5988f59ad06a" Oct 30 06:56:18.370223 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:18.370194 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjdmc" Oct 30 06:56:18.370566 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:18.370266 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:56:18.370566 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:18.370354 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:56:18.370566 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:18.370358 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjdmc" podUID="16e1a28b-a156-4d4e-820f-8712616b4eb2" Oct 30 06:56:18.370566 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:18.370452 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6vbv" podUID="f1b14112-6c97-4bfa-8f2a-5988f59ad06a" Oct 30 06:56:18.370566 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:18.370519 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts77g" podUID="96024683-dd4e-4d78-b5ad-6aa9ccc4bc05" Oct 30 06:56:18.583791 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:18.583716 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mjwrd" Oct 30 06:56:18.583925 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:18.583850 2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 06:56:18.584382 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:18.584362 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mjwrd" Oct 30 06:56:20.370100 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:20.370059 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjdmc" Oct 30 06:56:20.370604 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:20.370058 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:56:20.370604 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:20.370168 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjdmc" podUID="16e1a28b-a156-4d4e-820f-8712616b4eb2" Oct 30 06:56:20.370604 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:20.370230 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts77g" podUID="96024683-dd4e-4d78-b5ad-6aa9ccc4bc05" Oct 30 06:56:20.370604 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:20.370083 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:56:20.370604 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:20.370350 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6vbv" podUID="f1b14112-6c97-4bfa-8f2a-5988f59ad06a" Oct 30 06:56:22.370258 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.370224 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjdmc" Oct 30 06:56:22.370258 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.370251 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:56:22.370697 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.370273 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:56:22.370697 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:22.370413 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjdmc" podUID="16e1a28b-a156-4d4e-820f-8712616b4eb2" Oct 30 06:56:22.370789 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:22.370720 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts77g" podUID="96024683-dd4e-4d78-b5ad-6aa9ccc4bc05" Oct 30 06:56:22.370826 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:22.370793 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6vbv" podUID="f1b14112-6c97-4bfa-8f2a-5988f59ad06a" Oct 30 06:56:22.714171 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.714141 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeReady" Oct 30 06:56:22.714314 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.714268 2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 06:56:22.764341 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.764304 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7dwjw"] Oct 30 06:56:22.770025 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.769999 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-cjn9t"] Oct 30 06:56:22.770147 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.770117 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7dwjw" Oct 30 06:56:22.774024 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.773997 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5bl7s"] Oct 30 06:56:22.774167 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.774109 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-cjn9t" Oct 30 06:56:22.777232 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.777211 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zwfbr"] Oct 30 06:56:22.777337 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.777326 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5bl7s" Oct 30 06:56:22.780084 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.780062 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-l6qcv"] Oct 30 06:56:22.780429 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.780411 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-zwfbr" Oct 30 06:56:22.787387 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.787367 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 06:56:22.790478 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.790460 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 06:56:22.795631 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.795606 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-qqt8g"] Oct 30 06:56:22.795731 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.795719 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l6qcv" Oct 30 06:56:22.798097 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.796656 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 06:56:22.798097 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.796755 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-7pxn9\"" Oct 30 06:56:22.813918 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.813893 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7f6bb8cb6f-fzpwq"] Oct 30 06:56:22.814020 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.813989 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-qqt8g" Oct 30 06:56:22.835639 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.835604 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-zw4km"] Oct 30 06:56:22.835787 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.835667 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:22.851674 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.851634 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cflrf"] Oct 30 06:56:22.851801 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.851774 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" Oct 30 06:56:22.863963 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.863927 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8xtng"] Oct 30 06:56:22.864111 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.864045 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cflrf" Oct 30 06:56:22.876485 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.876454 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6q49\" (UniqueName: \"kubernetes.io/projected/14b28600-2ad3-47ff-a157-bcfacc6ab2ff-kube-api-access-z6q49\") pod \"kas-connection-checker-685895c7c9-7dwjw\" (UID: \"14b28600-2ad3-47ff-a157-bcfacc6ab2ff\") " pod="kube-system/kas-connection-checker-685895c7c9-7dwjw" Oct 30 06:56:22.876623 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.876487 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l6qcv\" (UID: \"8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l6qcv" Oct 30 06:56:22.876623 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.876563 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp7gc\" (UniqueName: \"kubernetes.io/projected/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-kube-api-access-gp7gc\") pod \"cluster-monitoring-operator-884576c57-l6qcv\" (UID: \"8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l6qcv" Oct 30 06:56:22.876695 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.876622 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5943f8dc-3e86-49fa-ac40-70a1f11fb69e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-cjn9t\" (UID: \"5943f8dc-3e86-49fa-ac40-70a1f11fb69e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cjn9t" Oct 30 06:56:22.876695 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.876658 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g5kc\" (UniqueName: \"kubernetes.io/projected/239195cf-db90-4e49-9064-8f7aa4af9b3a-kube-api-access-4g5kc\") pod \"kas-connection-checker-685895c7c9-zwfbr\" (UID: \"239195cf-db90-4e49-9064-8f7aa4af9b3a\") " pod="kube-system/kas-connection-checker-685895c7c9-zwfbr" Oct 30 06:56:22.876761 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.876694 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bth9\" (UniqueName: \"kubernetes.io/projected/8f93e69e-f48e-4c0a-ab3d-655739b6a8ae-kube-api-access-4bth9\") pod \"kas-connection-checker-685895c7c9-5bl7s\" (UID: \"8f93e69e-f48e-4c0a-ab3d-655739b6a8ae\") " pod="kube-system/kas-connection-checker-685895c7c9-5bl7s" Oct 30 06:56:22.876761 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.876748 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-l6qcv\" (UID: \"8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l6qcv" Oct 30 06:56:22.876816 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.876767 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5943f8dc-3e86-49fa-ac40-70a1f11fb69e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cjn9t\" (UID: \"5943f8dc-3e86-49fa-ac40-70a1f11fb69e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cjn9t" Oct 30 06:56:22.885993 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.885974 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5b5f74d469-87gxs"] Oct 30 06:56:22.886108 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.886093 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8xtng" Oct 30 06:56:22.887917 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.887899 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 06:56:22.887989 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.887949 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 06:56:22.888171 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.888158 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 06:56:22.888261 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.888246 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 06:56:22.888320 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.888300 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 06:56:22.888381 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.888343 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 06:56:22.888435 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.888394 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 06:56:22.888495 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.888436 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 06:56:22.888495 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.888466 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 06:56:22.888603 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.888508 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 06:56:22.888603 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.888546 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 06:56:22.888603 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.888553 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 06:56:22.888732 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.888627 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 06:56:22.888732 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.888683 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 06:56:22.900482 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.900466 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 06:56:22.901121 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.901105 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 06:56:22.908262 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.908246 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5dcb9bff4d-5xf9f"] Oct 30 06:56:22.908373 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.908357 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:22.913540 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.913520 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 06:56:22.913645 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.913603 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 06:56:22.913645 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.913624 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 06:56:22.913754 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.913726 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 06:56:22.913910 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.913787 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 06:56:22.913910 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.913839 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-f6r5d\"" Oct 30 06:56:22.913910 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.913848 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 06:56:22.913910 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.913779 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 06:56:22.913910 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.913876 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 06:56:22.913910 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.913891 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gp96c\"" Oct 30 06:56:22.914181 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.913955 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-lp7h7\"" Oct 30 06:56:22.914181 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.913974 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 06:56:22.914181 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.913988 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jgr8q\"" Oct 30 06:56:22.914181 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.914132 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 06:56:22.914299 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.914198 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-x2j6m\"" Oct 30 06:56:22.914299 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.914247 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-vcrml\"" Oct 30 06:56:22.922333 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.922315 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fv48w"] Oct 30 06:56:22.922412 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.922401 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:22.936564 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.936547 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-2kt77"] Oct 30 06:56:22.936651 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.936640 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fv48w" Oct 30 06:56:22.959217 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.959201 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z5nm2"] Oct 30 06:56:22.959328 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.959314 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2kt77" Oct 30 06:56:22.977306 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.977272 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-serving-cert\") pod \"insights-operator-7c4986d449-qqt8g\" (UID: \"4b59e668-d5a4-4ce0-a1e9-a16d63f51990\") " pod="openshift-insights/insights-operator-7c4986d449-qqt8g" Oct 30 06:56:22.977444 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.977310 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/20a9a026-75f0-4e90-b66a-066d3b80facf-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-8xtng\" (UID: \"20a9a026-75f0-4e90-b66a-066d3b80facf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8xtng" Oct 30 06:56:22.977444 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.977391 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/50ab5bd7-ea08-460b-9932-1dac54f3005b-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-cflrf\" (UID: \"50ab5bd7-ea08-460b-9932-1dac54f3005b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cflrf" Oct 30 06:56:22.977444 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.977431 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6q49\" (UniqueName: \"kubernetes.io/projected/14b28600-2ad3-47ff-a157-bcfacc6ab2ff-kube-api-access-z6q49\") pod \"kas-connection-checker-685895c7c9-7dwjw\" (UID: \"14b28600-2ad3-47ff-a157-bcfacc6ab2ff\") " pod="kube-system/kas-connection-checker-685895c7c9-7dwjw" Oct 30 06:56:22.977540 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.977452 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l6qcv\" (UID: \"8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l6qcv" Oct 30 06:56:22.977540 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.977468 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp7gc\" (UniqueName: \"kubernetes.io/projected/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-kube-api-access-gp7gc\") pod \"cluster-monitoring-operator-884576c57-l6qcv\" (UID: \"8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l6qcv" Oct 30 06:56:22.977540 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.977491 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-service-ca-bundle\") pod \"insights-operator-7c4986d449-qqt8g\" (UID: \"4b59e668-d5a4-4ce0-a1e9-a16d63f51990\") " pod="openshift-insights/insights-operator-7c4986d449-qqt8g" Oct 30 06:56:22.977540 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.977520 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5943f8dc-3e86-49fa-ac40-70a1f11fb69e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-cjn9t\" (UID: \"5943f8dc-3e86-49fa-ac40-70a1f11fb69e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cjn9t" Oct 30 06:56:22.977741 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.977545 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/42802760-36a8-4b79-8690-5a00528503c7-config\") pod \"console-operator-7cd56b8dbc-zw4km\" (UID: \"42802760-36a8-4b79-8690-5a00528503c7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" Oct 30 06:56:22.977741 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:22.977585 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 06:56:22.977741 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:22.977637 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-cluster-monitoring-operator-tls podName:8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:23.477623211 +0000 UTC m=+30.606759707 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-l6qcv" (UID: "8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2") : secret "cluster-monitoring-operator-tls" not found Oct 30 06:56:22.977741 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.977651 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42802760-36a8-4b79-8690-5a00528503c7-trusted-ca\") pod \"console-operator-7cd56b8dbc-zw4km\" (UID: \"42802760-36a8-4b79-8690-5a00528503c7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" Oct 30 06:56:22.977741 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.977670 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/20a9a026-75f0-4e90-b66a-066d3b80facf-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-8xtng\" (UID: \"20a9a026-75f0-4e90-b66a-066d3b80facf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8xtng" Oct 30 06:56:22.977741 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.977706 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-service-ca-bundle\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:22.978022 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.977754 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-l6qcv\" (UID: \"8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l6qcv" Oct 30 06:56:22.978022 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.977800 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5943f8dc-3e86-49fa-ac40-70a1f11fb69e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cjn9t\" (UID: \"5943f8dc-3e86-49fa-ac40-70a1f11fb69e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cjn9t" Oct 30 06:56:22.978022 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.977850 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h86bq\" (UniqueName: \"kubernetes.io/projected/50ab5bd7-ea08-460b-9932-1dac54f3005b-kube-api-access-h86bq\") pod \"cluster-samples-operator-6ddb6c8b94-cflrf\" (UID: \"50ab5bd7-ea08-460b-9932-1dac54f3005b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cflrf" Oct 30 06:56:22.978022 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.977885 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-default-certificate\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:22.978022 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.977918 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbpxj\" (UniqueName: \"kubernetes.io/projected/42802760-36a8-4b79-8690-5a00528503c7-kube-api-access-zbpxj\") pod \"console-operator-7cd56b8dbc-zw4km\" (UID: \"42802760-36a8-4b79-8690-5a00528503c7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" Oct 30 06:56:22.978022 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.977943 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9n88\" (UniqueName: \"kubernetes.io/projected/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-kube-api-access-z9n88\") pod \"insights-operator-7c4986d449-qqt8g\" (UID: \"4b59e668-d5a4-4ce0-a1e9-a16d63f51990\") " pod="openshift-insights/insights-operator-7c4986d449-qqt8g" Oct 30 06:56:22.978022 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.977973 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-snapshots\") pod \"insights-operator-7c4986d449-qqt8g\" (UID: \"4b59e668-d5a4-4ce0-a1e9-a16d63f51990\") " pod="openshift-insights/insights-operator-7c4986d449-qqt8g" Oct 30 06:56:22.978022 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.977998 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-qqt8g\" (UID: \"4b59e668-d5a4-4ce0-a1e9-a16d63f51990\") " pod="openshift-insights/insights-operator-7c4986d449-qqt8g" Oct 30 06:56:22.978022 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.978027 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-metrics-certs\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:22.978499 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.978052 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-stats-auth\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:22.978499 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.978082 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g5kc\" (UniqueName: \"kubernetes.io/projected/239195cf-db90-4e49-9064-8f7aa4af9b3a-kube-api-access-4g5kc\") pod \"kas-connection-checker-685895c7c9-zwfbr\" (UID: \"239195cf-db90-4e49-9064-8f7aa4af9b3a\") " pod="kube-system/kas-connection-checker-685895c7c9-zwfbr" Oct 30 06:56:22.978499 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.978111 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bth9\" (UniqueName: \"kubernetes.io/projected/8f93e69e-f48e-4c0a-ab3d-655739b6a8ae-kube-api-access-4bth9\") pod \"kas-connection-checker-685895c7c9-5bl7s\" (UID: \"8f93e69e-f48e-4c0a-ab3d-655739b6a8ae\") " pod="kube-system/kas-connection-checker-685895c7c9-5bl7s" Oct 30 06:56:22.978499 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.978146 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-tmp\") pod \"insights-operator-7c4986d449-qqt8g\" (UID: \"4b59e668-d5a4-4ce0-a1e9-a16d63f51990\") " pod="openshift-insights/insights-operator-7c4986d449-qqt8g" Oct 30 06:56:22.978499 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.978172 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vrh9\" (UniqueName: \"kubernetes.io/projected/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-kube-api-access-4vrh9\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:22.978499 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.978209 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67hxp\" (UniqueName: \"kubernetes.io/projected/20a9a026-75f0-4e90-b66a-066d3b80facf-kube-api-access-67hxp\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-8xtng\" (UID: \"20a9a026-75f0-4e90-b66a-066d3b80facf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8xtng" Oct 30 06:56:22.978499 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.978244 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/42802760-36a8-4b79-8690-5a00528503c7-serving-cert\") pod \"console-operator-7cd56b8dbc-zw4km\" (UID: \"42802760-36a8-4b79-8690-5a00528503c7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" Oct 30 06:56:22.978499 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.978266 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/50ab5bd7-ea08-460b-9932-1dac54f3005b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-cflrf\" (UID: \"50ab5bd7-ea08-460b-9932-1dac54f3005b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cflrf" Oct 30 06:56:22.978499 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:22.978395 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 06:56:22.978499 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:22.978453 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5943f8dc-3e86-49fa-ac40-70a1f11fb69e-networking-console-plugin-cert podName:5943f8dc-3e86-49fa-ac40-70a1f11fb69e nodeName:}" failed. No retries permitted until 2026-10-30 06:56:23.478437165 +0000 UTC m=+30.607573691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5943f8dc-3e86-49fa-ac40-70a1f11fb69e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-cjn9t" (UID: "5943f8dc-3e86-49fa-ac40-70a1f11fb69e") : secret "networking-console-plugin-cert" not found Oct 30 06:56:22.978499 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.978392 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5943f8dc-3e86-49fa-ac40-70a1f11fb69e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-cjn9t\" (UID: \"5943f8dc-3e86-49fa-ac40-70a1f11fb69e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cjn9t" Oct 30 06:56:22.978906 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.978557 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-l6qcv\" (UID: \"8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l6qcv" Oct 30 06:56:22.985447 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.985421 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vgrqz"] Oct 30 06:56:22.985560 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.985544 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z5nm2" Oct 30 06:56:22.988055 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.988031 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 06:56:22.988164 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.988051 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 06:56:22.988164 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.988059 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 06:56:22.988164 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.988052 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 06:56:22.988164 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.988095 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 06:56:22.988164 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.988125 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 06:56:22.988340 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.988190 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 06:56:22.988340 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.988214 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 06:56:22.988814 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.988794 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 06:56:22.995386 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.995367 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 06:56:22.995386 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.995379 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 06:56:22.995687 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.995423 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 06:56:22.995687 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.995493 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-b2dtw\"" Oct 30 06:56:22.995687 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.995588 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-mkfxw\"" Oct 30 06:56:22.995687 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.995641 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7cszt\"" Oct 30 06:56:22.995876 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:22.995693 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 06:56:23.005327 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.005294 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6q49\" (UniqueName: \"kubernetes.io/projected/14b28600-2ad3-47ff-a157-bcfacc6ab2ff-kube-api-access-z6q49\") pod \"kas-connection-checker-685895c7c9-7dwjw\" (UID: \"14b28600-2ad3-47ff-a157-bcfacc6ab2ff\") " pod="kube-system/kas-connection-checker-685895c7c9-7dwjw" Oct 30 06:56:23.005327 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.005324 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qrksf"] Oct 30 06:56:23.005518 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.005339 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g5kc\" (UniqueName: \"kubernetes.io/projected/239195cf-db90-4e49-9064-8f7aa4af9b3a-kube-api-access-4g5kc\") pod \"kas-connection-checker-685895c7c9-zwfbr\" (UID: \"239195cf-db90-4e49-9064-8f7aa4af9b3a\") " pod="kube-system/kas-connection-checker-685895c7c9-zwfbr" Oct 30 06:56:23.005518 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.005372 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bth9\" (UniqueName: \"kubernetes.io/projected/8f93e69e-f48e-4c0a-ab3d-655739b6a8ae-kube-api-access-4bth9\") pod \"kas-connection-checker-685895c7c9-5bl7s\" (UID: \"8f93e69e-f48e-4c0a-ab3d-655739b6a8ae\") " pod="kube-system/kas-connection-checker-685895c7c9-5bl7s" Oct 30 06:56:23.005518 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.005437 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vgrqz" Oct 30 06:56:23.005518 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.005440 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp7gc\" (UniqueName: \"kubernetes.io/projected/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-kube-api-access-gp7gc\") pod \"cluster-monitoring-operator-884576c57-l6qcv\" (UID: \"8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l6qcv" Oct 30 06:56:23.022795 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.022771 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7dwjw"] Oct 30 06:56:23.022922 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.022811 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5bl7s"] Oct 30 06:56:23.022922 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.022835 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zwfbr"] Oct 30 06:56:23.022922 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.022857 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-zw4km"] Oct 30 06:56:23.022922 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.022877 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-qqt8g"] Oct 30 06:56:23.022922 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.022886 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qrksf" Oct 30 06:56:23.022922 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.022895 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cflrf"] Oct 30 06:56:23.022922 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.022924 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-l6qcv"] Oct 30 06:56:23.023146 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.022943 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8xtng"] Oct 30 06:56:23.023146 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.022965 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-2kt77"] Oct 30 06:56:23.023146 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.022988 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-cjn9t"] Oct 30 06:56:23.023146 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.023013 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z5nm2"] Oct 30 06:56:23.023146 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.023034 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fv48w"] Oct 30 06:56:23.023146 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.023055 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qrksf"] Oct 30 06:56:23.023146 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.023075 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5dcb9bff4d-5xf9f"] Oct 30 06:56:23.023146 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.023095 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7f6bb8cb6f-fzpwq"] Oct 30 06:56:23.023146 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.023114 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b5f74d469-87gxs"] Oct 30 06:56:23.023146 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.023134 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vgrqz"] Oct 30 06:56:23.079523 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.079488 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/36b00a60-14fb-4364-8943-6bcacc5365cc-trusted-ca\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:23.079694 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.079532 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-snapshots\") pod \"insights-operator-7c4986d449-qqt8g\" (UID: \"4b59e668-d5a4-4ce0-a1e9-a16d63f51990\") " pod="openshift-insights/insights-operator-7c4986d449-qqt8g" Oct 30 06:56:23.079694 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.079641 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-qqt8g\" (UID: \"4b59e668-d5a4-4ce0-a1e9-a16d63f51990\") " pod="openshift-insights/insights-operator-7c4986d449-qqt8g" Oct 30 06:56:23.079694 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.079676 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-metrics-certs\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:23.079861 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.079696 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c5f1f83f-9bfd-45f5-b065-ba95cf659956-serving-cert\") pod \"service-ca-operator-544d749c8c-2kt77\" (UID: \"c5f1f83f-9bfd-45f5-b065-ba95cf659956\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2kt77" Oct 30 06:56:23.079861 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.079721 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9d1faf4-274f-49c8-9340-f85463840b27-trusted-ca\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:23.079861 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:23.079785 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 06:56:23.079861 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.079822 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-stats-auth\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:23.079861 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:23.079842 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-metrics-certs podName:0c49a4a4-033f-4a62-8bc4-6b59531e8bf7 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:23.579820916 +0000 UTC m=+30.708957421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-metrics-certs") pod "router-default-7f6bb8cb6f-fzpwq" (UID: "0c49a4a4-033f-4a62-8bc4-6b59531e8bf7") : secret "router-metrics-certs-default" not found Oct 30 06:56:23.080110 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.079873 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s88rb\" (UniqueName: \"kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-kube-api-access-s88rb\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:23.080110 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.079913 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-tmp\") pod \"insights-operator-7c4986d449-qqt8g\" (UID: \"4b59e668-d5a4-4ce0-a1e9-a16d63f51990\") " pod="openshift-insights/insights-operator-7c4986d449-qqt8g" Oct 30 06:56:23.080110 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.079939 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vrh9\" (UniqueName: \"kubernetes.io/projected/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-kube-api-access-4vrh9\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:23.080110 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.079982 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrc6h\" (UniqueName: \"kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-kube-api-access-rrc6h\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:23.080110 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.080030 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67hxp\" (UniqueName: \"kubernetes.io/projected/20a9a026-75f0-4e90-b66a-066d3b80facf-kube-api-access-67hxp\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-8xtng\" (UID: \"20a9a026-75f0-4e90-b66a-066d3b80facf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8xtng" Oct 30 06:56:23.080356 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.080149 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9d1faf4-274f-49c8-9340-f85463840b27-installation-pull-secrets\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:23.080356 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.080176 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-snapshots\") pod \"insights-operator-7c4986d449-qqt8g\" (UID: \"4b59e668-d5a4-4ce0-a1e9-a16d63f51990\") " pod="openshift-insights/insights-operator-7c4986d449-qqt8g" Oct 30 06:56:23.080356 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.080216 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm5vg\" (UniqueName: \"kubernetes.io/projected/c5f1f83f-9bfd-45f5-b065-ba95cf659956-kube-api-access-qm5vg\") pod \"service-ca-operator-544d749c8c-2kt77\" (UID: \"c5f1f83f-9bfd-45f5-b065-ba95cf659956\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2kt77" Oct 30 06:56:23.080356 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.080268 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-tmp\") pod \"insights-operator-7c4986d449-qqt8g\" (UID: \"4b59e668-d5a4-4ce0-a1e9-a16d63f51990\") " pod="openshift-insights/insights-operator-7c4986d449-qqt8g" Oct 30 06:56:23.080356 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.080268 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/42802760-36a8-4b79-8690-5a00528503c7-serving-cert\") pod \"console-operator-7cd56b8dbc-zw4km\" (UID: \"42802760-36a8-4b79-8690-5a00528503c7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" Oct 30 06:56:23.080356 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.080319 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/50ab5bd7-ea08-460b-9932-1dac54f3005b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-cflrf\" (UID: \"50ab5bd7-ea08-460b-9932-1dac54f3005b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cflrf" Oct 30 06:56:23.080356 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.080348 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-serving-cert\") pod \"insights-operator-7c4986d449-qqt8g\" (UID: \"4b59e668-d5a4-4ce0-a1e9-a16d63f51990\") " pod="openshift-insights/insights-operator-7c4986d449-qqt8g" Oct 30 06:56:23.080718 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.080379 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c5f1f83f-9bfd-45f5-b065-ba95cf659956-config\") pod \"service-ca-operator-544d749c8c-2kt77\" (UID: \"c5f1f83f-9bfd-45f5-b065-ba95cf659956\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2kt77" Oct 30 06:56:23.080718 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.080404 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-984ts\" (UniqueName: \"kubernetes.io/projected/584e4eb5-6e5e-4f96-b97a-eb58fd762347-kube-api-access-984ts\") pod \"volume-data-source-validator-58c9f5f46d-z5nm2\" (UID: \"584e4eb5-6e5e-4f96-b97a-eb58fd762347\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z5nm2" Oct 30 06:56:23.080718 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:23.080415 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 06:56:23.080718 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.080433 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/20a9a026-75f0-4e90-b66a-066d3b80facf-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-8xtng\" (UID: \"20a9a026-75f0-4e90-b66a-066d3b80facf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8xtng" Oct 30 06:56:23.080718 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:23.080461 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50ab5bd7-ea08-460b-9932-1dac54f3005b-samples-operator-tls podName:50ab5bd7-ea08-460b-9932-1dac54f3005b nodeName:}" failed. No retries permitted until 2026-10-30 06:56:23.580444905 +0000 UTC m=+30.709581397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/50ab5bd7-ea08-460b-9932-1dac54f3005b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-cflrf" (UID: "50ab5bd7-ea08-460b-9932-1dac54f3005b") : secret "samples-operator-tls" not found Oct 30 06:56:23.080718 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.080487 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-945vk\" (UniqueName: \"kubernetes.io/projected/35296121-39f4-4fae-ae2f-f92790ad3f5c-kube-api-access-945vk\") pod \"network-check-source-6f78d59f99-fv48w\" (UID: \"35296121-39f4-4fae-ae2f-f92790ad3f5c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fv48w" Oct 30 06:56:23.080718 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.080514 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9d1faf4-274f-49c8-9340-f85463840b27-image-registry-private-configuration\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:23.080718 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.080540 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:23.080718 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.080549 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-qqt8g\" (UID: \"4b59e668-d5a4-4ce0-a1e9-a16d63f51990\") " pod="openshift-insights/insights-operator-7c4986d449-qqt8g" Oct 30 06:56:23.080718 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.080608 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/50ab5bd7-ea08-460b-9932-1dac54f3005b-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-cflrf\" (UID: \"50ab5bd7-ea08-460b-9932-1dac54f3005b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cflrf" Oct 30 06:56:23.080718 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.080635 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:23.081249 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.080815 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-service-ca-bundle\") pod \"insights-operator-7c4986d449-qqt8g\" (UID: \"4b59e668-d5a4-4ce0-a1e9-a16d63f51990\") " pod="openshift-insights/insights-operator-7c4986d449-qqt8g" Oct 30 06:56:23.081249 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.080852 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9d1faf4-274f-49c8-9340-f85463840b27-registry-certificates\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:23.081249 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.080884 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-bound-sa-token\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:23.081249 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.080915 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/42802760-36a8-4b79-8690-5a00528503c7-config\") pod \"console-operator-7cd56b8dbc-zw4km\" (UID: \"42802760-36a8-4b79-8690-5a00528503c7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" Oct 30 06:56:23.081249 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.080943 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/36b00a60-14fb-4364-8943-6bcacc5365cc-installation-pull-secrets\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:23.081249 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.080971 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/36b00a60-14fb-4364-8943-6bcacc5365cc-image-registry-private-configuration\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:23.081249 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.080992 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/20a9a026-75f0-4e90-b66a-066d3b80facf-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-8xtng\" (UID: \"20a9a026-75f0-4e90-b66a-066d3b80facf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8xtng" Oct 30 06:56:23.081249 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.081009 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-bound-sa-token\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:23.081249 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.081052 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42802760-36a8-4b79-8690-5a00528503c7-trusted-ca\") pod \"console-operator-7cd56b8dbc-zw4km\" (UID: \"42802760-36a8-4b79-8690-5a00528503c7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" Oct 30 06:56:23.081249 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.081082 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/20a9a026-75f0-4e90-b66a-066d3b80facf-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-8xtng\" (UID: \"20a9a026-75f0-4e90-b66a-066d3b80facf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8xtng" Oct 30 06:56:23.081249 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.081106 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/36b00a60-14fb-4364-8943-6bcacc5365cc-ca-trust-extracted\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:23.081249 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.081132 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-service-ca-bundle\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:23.081249 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.081162 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9d1faf4-274f-49c8-9340-f85463840b27-ca-trust-extracted\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:23.081249 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.081202 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h86bq\" (UniqueName: \"kubernetes.io/projected/50ab5bd7-ea08-460b-9932-1dac54f3005b-kube-api-access-h86bq\") pod \"cluster-samples-operator-6ddb6c8b94-cflrf\" (UID: \"50ab5bd7-ea08-460b-9932-1dac54f3005b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cflrf" Oct 30 06:56:23.081249 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.081227 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-default-certificate\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:23.081968 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.081255 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-certificates\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:23.081968 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.081285 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbpxj\" (UniqueName: \"kubernetes.io/projected/42802760-36a8-4b79-8690-5a00528503c7-kube-api-access-zbpxj\") pod \"console-operator-7cd56b8dbc-zw4km\" (UID: \"42802760-36a8-4b79-8690-5a00528503c7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" Oct 30 06:56:23.081968 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.081308 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9n88\" (UniqueName: \"kubernetes.io/projected/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-kube-api-access-z9n88\") pod \"insights-operator-7c4986d449-qqt8g\" (UID: \"4b59e668-d5a4-4ce0-a1e9-a16d63f51990\") " pod="openshift-insights/insights-operator-7c4986d449-qqt8g" Oct 30 06:56:23.081968 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.081326 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-service-ca-bundle\") pod \"insights-operator-7c4986d449-qqt8g\" (UID: \"4b59e668-d5a4-4ce0-a1e9-a16d63f51990\") " pod="openshift-insights/insights-operator-7c4986d449-qqt8g" Oct 30 06:56:23.081968 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.081393 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/50ab5bd7-ea08-460b-9932-1dac54f3005b-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-cflrf\" (UID: \"50ab5bd7-ea08-460b-9932-1dac54f3005b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cflrf" Oct 30 06:56:23.081968 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.081690 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/42802760-36a8-4b79-8690-5a00528503c7-config\") pod \"console-operator-7cd56b8dbc-zw4km\" (UID: \"42802760-36a8-4b79-8690-5a00528503c7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" Oct 30 06:56:23.081968 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:23.081787 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-service-ca-bundle podName:0c49a4a4-033f-4a62-8bc4-6b59531e8bf7 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:23.581769997 +0000 UTC m=+30.710906505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-service-ca-bundle") pod "router-default-7f6bb8cb6f-fzpwq" (UID: "0c49a4a4-033f-4a62-8bc4-6b59531e8bf7") : configmap references non-existent config key: service-ca.crt Oct 30 06:56:23.082396 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.082374 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42802760-36a8-4b79-8690-5a00528503c7-trusted-ca\") pod \"console-operator-7cd56b8dbc-zw4km\" (UID: \"42802760-36a8-4b79-8690-5a00528503c7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" Oct 30 06:56:23.082735 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.082716 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-stats-auth\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:23.083147 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.083126 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-serving-cert\") pod \"insights-operator-7c4986d449-qqt8g\" (UID: \"4b59e668-d5a4-4ce0-a1e9-a16d63f51990\") " pod="openshift-insights/insights-operator-7c4986d449-qqt8g" Oct 30 06:56:23.083424 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.083408 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/42802760-36a8-4b79-8690-5a00528503c7-serving-cert\") pod \"console-operator-7cd56b8dbc-zw4km\" (UID: \"42802760-36a8-4b79-8690-5a00528503c7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" Oct 30 06:56:23.084097 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.084072 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-default-certificate\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:23.084186 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.084163 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/20a9a026-75f0-4e90-b66a-066d3b80facf-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-8xtng\" (UID: \"20a9a026-75f0-4e90-b66a-066d3b80facf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8xtng" Oct 30 06:56:23.087730 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.087710 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7dwjw" Oct 30 06:56:23.088084 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.088063 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 06:56:23.088084 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.088063 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 06:56:23.088084 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.088069 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 06:56:23.088084 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.088111 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 06:56:23.088449 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.088153 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 06:56:23.095548 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.095519 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 06:56:23.095679 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.095610 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 06:56:23.095745 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.095699 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-25rrt\"" Oct 30 06:56:23.095745 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.095709 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r9nbg\"" Oct 30 06:56:23.095745 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.095718 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-4flrq\"" Oct 30 06:56:23.103005 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.102980 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vrh9\" (UniqueName: \"kubernetes.io/projected/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-kube-api-access-4vrh9\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:23.103098 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.103007 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67hxp\" (UniqueName: \"kubernetes.io/projected/20a9a026-75f0-4e90-b66a-066d3b80facf-kube-api-access-67hxp\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-8xtng\" (UID: \"20a9a026-75f0-4e90-b66a-066d3b80facf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8xtng" Oct 30 06:56:23.104036 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.104011 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9n88\" (UniqueName: \"kubernetes.io/projected/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-kube-api-access-z9n88\") pod \"insights-operator-7c4986d449-qqt8g\" (UID: \"4b59e668-d5a4-4ce0-a1e9-a16d63f51990\") " pod="openshift-insights/insights-operator-7c4986d449-qqt8g" Oct 30 06:56:23.104351 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.104332 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h86bq\" (UniqueName: \"kubernetes.io/projected/50ab5bd7-ea08-460b-9932-1dac54f3005b-kube-api-access-h86bq\") pod \"cluster-samples-operator-6ddb6c8b94-cflrf\" (UID: \"50ab5bd7-ea08-460b-9932-1dac54f3005b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cflrf" Oct 30 06:56:23.119748 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.119701 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbpxj\" (UniqueName: \"kubernetes.io/projected/42802760-36a8-4b79-8690-5a00528503c7-kube-api-access-zbpxj\") pod \"console-operator-7cd56b8dbc-zw4km\" (UID: \"42802760-36a8-4b79-8690-5a00528503c7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" Oct 30 06:56:23.122070 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.122054 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5bl7s" Oct 30 06:56:23.146839 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.146812 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-zwfbr" Oct 30 06:56:23.160828 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.160762 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-qqt8g" Oct 30 06:56:23.182143 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.181891 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d930accb-e4d8-404b-8155-1b6a8edd812e-tmp-dir\") pod \"dns-default-vgrqz\" (UID: \"d930accb-e4d8-404b-8155-1b6a8edd812e\") " pod="openshift-dns/dns-default-vgrqz" Oct 30 06:56:23.182143 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.181940 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/36b00a60-14fb-4364-8943-6bcacc5365cc-trusted-ca\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:23.182143 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.182045 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ab9750b-0582-4c7c-b770-b1b7fbe8994c-cert\") pod \"ingress-canary-qrksf\" (UID: \"1ab9750b-0582-4c7c-b770-b1b7fbe8994c\") " pod="openshift-ingress-canary/ingress-canary-qrksf" Oct 30 06:56:23.182143 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.182104 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c5f1f83f-9bfd-45f5-b065-ba95cf659956-serving-cert\") pod \"service-ca-operator-544d749c8c-2kt77\" (UID: \"c5f1f83f-9bfd-45f5-b065-ba95cf659956\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2kt77" Oct 30 06:56:23.182352 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.182155 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9d1faf4-274f-49c8-9340-f85463840b27-trusted-ca\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:23.182990 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.182445 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s88rb\" (UniqueName: \"kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-kube-api-access-s88rb\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:23.182990 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.182486 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrc6h\" (UniqueName: \"kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-kube-api-access-rrc6h\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:23.182990 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.182513 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9d1faf4-274f-49c8-9340-f85463840b27-installation-pull-secrets\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:23.182990 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.182531 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm5vg\" (UniqueName: \"kubernetes.io/projected/c5f1f83f-9bfd-45f5-b065-ba95cf659956-kube-api-access-qm5vg\") pod \"service-ca-operator-544d749c8c-2kt77\" (UID: \"c5f1f83f-9bfd-45f5-b065-ba95cf659956\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2kt77" Oct 30 06:56:23.182990 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.182560 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c5f1f83f-9bfd-45f5-b065-ba95cf659956-config\") pod \"service-ca-operator-544d749c8c-2kt77\" (UID: \"c5f1f83f-9bfd-45f5-b065-ba95cf659956\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2kt77" Oct 30 06:56:23.182990 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.182595 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-984ts\" (UniqueName: \"kubernetes.io/projected/584e4eb5-6e5e-4f96-b97a-eb58fd762347-kube-api-access-984ts\") pod \"volume-data-source-validator-58c9f5f46d-z5nm2\" (UID: \"584e4eb5-6e5e-4f96-b97a-eb58fd762347\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z5nm2" Oct 30 06:56:23.182990 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.182618 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-945vk\" (UniqueName: \"kubernetes.io/projected/35296121-39f4-4fae-ae2f-f92790ad3f5c-kube-api-access-945vk\") pod \"network-check-source-6f78d59f99-fv48w\" (UID: \"35296121-39f4-4fae-ae2f-f92790ad3f5c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fv48w" Oct 30 06:56:23.182990 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.182633 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9d1faf4-274f-49c8-9340-f85463840b27-image-registry-private-configuration\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:23.182990 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.182650 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:23.182990 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.182684 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:23.182990 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.182711 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d930accb-e4d8-404b-8155-1b6a8edd812e-config-volume\") pod \"dns-default-vgrqz\" (UID: \"d930accb-e4d8-404b-8155-1b6a8edd812e\") " pod="openshift-dns/dns-default-vgrqz" Oct 30 06:56:23.182990 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.182731 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9d1faf4-274f-49c8-9340-f85463840b27-registry-certificates\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:23.182990 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.182745 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-bound-sa-token\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:23.182990 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.182763 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx6zb\" (UniqueName: \"kubernetes.io/projected/1ab9750b-0582-4c7c-b770-b1b7fbe8994c-kube-api-access-jx6zb\") pod \"ingress-canary-qrksf\" (UID: \"1ab9750b-0582-4c7c-b770-b1b7fbe8994c\") " pod="openshift-ingress-canary/ingress-canary-qrksf" Oct 30 06:56:23.182990 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.182781 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d930accb-e4d8-404b-8155-1b6a8edd812e-metrics-tls\") pod \"dns-default-vgrqz\" (UID: \"d930accb-e4d8-404b-8155-1b6a8edd812e\") " pod="openshift-dns/dns-default-vgrqz" Oct 30 06:56:23.182990 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:23.182881 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 06:56:23.183819 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.182897 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/36b00a60-14fb-4364-8943-6bcacc5365cc-installation-pull-secrets\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:23.183819 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.182919 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgbhv\" (UniqueName: \"kubernetes.io/projected/d930accb-e4d8-404b-8155-1b6a8edd812e-kube-api-access-lgbhv\") pod \"dns-default-vgrqz\" (UID: \"d930accb-e4d8-404b-8155-1b6a8edd812e\") " pod="openshift-dns/dns-default-vgrqz" Oct 30 06:56:23.183819 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.182938 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/36b00a60-14fb-4364-8943-6bcacc5365cc-image-registry-private-configuration\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:23.183819 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.182956 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-bound-sa-token\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:23.183819 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.182967 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/36b00a60-14fb-4364-8943-6bcacc5365cc-trusted-ca\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:23.183819 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.183000 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/36b00a60-14fb-4364-8943-6bcacc5365cc-ca-trust-extracted\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:23.183819 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.183040 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9d1faf4-274f-49c8-9340-f85463840b27-ca-trust-extracted\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:23.183819 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.183090 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-certificates\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:23.183819 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.183529 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c5f1f83f-9bfd-45f5-b065-ba95cf659956-config\") pod \"service-ca-operator-544d749c8c-2kt77\" (UID: \"c5f1f83f-9bfd-45f5-b065-ba95cf659956\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2kt77" Oct 30 06:56:23.183819 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.183643 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-certificates\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:23.183819 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:23.182900 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b5f74d469-87gxs: secret "image-registry-tls" not found Oct 30 06:56:23.183819 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.183704 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9d1faf4-274f-49c8-9340-f85463840b27-registry-certificates\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:23.183819 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:23.183723 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls podName:b9d1faf4-274f-49c8-9340-f85463840b27 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:23.683703912 +0000 UTC m=+30.812840404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls") pod "image-registry-5b5f74d469-87gxs" (UID: "b9d1faf4-274f-49c8-9340-f85463840b27") : secret "image-registry-tls" not found Oct 30 06:56:23.183819 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.183784 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9d1faf4-274f-49c8-9340-f85463840b27-trusted-ca\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:23.184370 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:23.183069 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 06:56:23.184370 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:23.183856 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dcb9bff4d-5xf9f: secret "image-registry-tls" not found Oct 30 06:56:23.184370 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:23.183945 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls podName:36b00a60-14fb-4364-8943-6bcacc5365cc nodeName:}" failed. No retries permitted until 2026-10-30 06:56:23.683923068 +0000 UTC m=+30.813059575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls") pod "image-registry-5dcb9bff4d-5xf9f" (UID: "36b00a60-14fb-4364-8943-6bcacc5365cc") : secret "image-registry-tls" not found Oct 30 06:56:23.184370 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.184083 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/36b00a60-14fb-4364-8943-6bcacc5365cc-ca-trust-extracted\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:23.184370 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.184251 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9d1faf4-274f-49c8-9340-f85463840b27-ca-trust-extracted\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:23.184676 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.184658 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c5f1f83f-9bfd-45f5-b065-ba95cf659956-serving-cert\") pod \"service-ca-operator-544d749c8c-2kt77\" (UID: \"c5f1f83f-9bfd-45f5-b065-ba95cf659956\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2kt77" Oct 30 06:56:23.185699 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.185678 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9d1faf4-274f-49c8-9340-f85463840b27-image-registry-private-configuration\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:23.186115 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.186092 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/36b00a60-14fb-4364-8943-6bcacc5365cc-installation-pull-secrets\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:23.186235 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.186216 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/36b00a60-14fb-4364-8943-6bcacc5365cc-image-registry-private-configuration\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:23.187403 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.187380 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9d1faf4-274f-49c8-9340-f85463840b27-installation-pull-secrets\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:23.209221 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.209142 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s88rb\" (UniqueName: \"kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-kube-api-access-s88rb\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:23.209471 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.209418 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-984ts\" (UniqueName: \"kubernetes.io/projected/584e4eb5-6e5e-4f96-b97a-eb58fd762347-kube-api-access-984ts\") pod \"volume-data-source-validator-58c9f5f46d-z5nm2\" (UID: \"584e4eb5-6e5e-4f96-b97a-eb58fd762347\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z5nm2" Oct 30 06:56:23.210585 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.210523 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-bound-sa-token\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:23.210725 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.210709 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrc6h\" (UniqueName: \"kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-kube-api-access-rrc6h\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:23.210831 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.210804 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-945vk\" (UniqueName: \"kubernetes.io/projected/35296121-39f4-4fae-ae2f-f92790ad3f5c-kube-api-access-945vk\") pod \"network-check-source-6f78d59f99-fv48w\" (UID: \"35296121-39f4-4fae-ae2f-f92790ad3f5c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fv48w" Oct 30 06:56:23.212621 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.212143 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm5vg\" (UniqueName: \"kubernetes.io/projected/c5f1f83f-9bfd-45f5-b065-ba95cf659956-kube-api-access-qm5vg\") pod \"service-ca-operator-544d749c8c-2kt77\" (UID: \"c5f1f83f-9bfd-45f5-b065-ba95cf659956\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2kt77" Oct 30 06:56:23.213013 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.212988 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-bound-sa-token\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:23.218590 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.218555 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8xtng" Oct 30 06:56:23.233621 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.233595 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" Oct 30 06:56:23.288722 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.286947 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d930accb-e4d8-404b-8155-1b6a8edd812e-config-volume\") pod \"dns-default-vgrqz\" (UID: \"d930accb-e4d8-404b-8155-1b6a8edd812e\") " pod="openshift-dns/dns-default-vgrqz" Oct 30 06:56:23.288722 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.287010 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx6zb\" (UniqueName: \"kubernetes.io/projected/1ab9750b-0582-4c7c-b770-b1b7fbe8994c-kube-api-access-jx6zb\") pod \"ingress-canary-qrksf\" (UID: \"1ab9750b-0582-4c7c-b770-b1b7fbe8994c\") " pod="openshift-ingress-canary/ingress-canary-qrksf" Oct 30 06:56:23.288722 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.287052 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d930accb-e4d8-404b-8155-1b6a8edd812e-metrics-tls\") pod \"dns-default-vgrqz\" (UID: \"d930accb-e4d8-404b-8155-1b6a8edd812e\") " pod="openshift-dns/dns-default-vgrqz" Oct 30 06:56:23.288722 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.287091 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgbhv\" (UniqueName: \"kubernetes.io/projected/d930accb-e4d8-404b-8155-1b6a8edd812e-kube-api-access-lgbhv\") pod \"dns-default-vgrqz\" (UID: \"d930accb-e4d8-404b-8155-1b6a8edd812e\") " pod="openshift-dns/dns-default-vgrqz" Oct 30 06:56:23.288722 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.287202 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d930accb-e4d8-404b-8155-1b6a8edd812e-tmp-dir\") pod \"dns-default-vgrqz\" (UID: \"d930accb-e4d8-404b-8155-1b6a8edd812e\") " pod="openshift-dns/dns-default-vgrqz" Oct 30 06:56:23.288722 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.287243 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ab9750b-0582-4c7c-b770-b1b7fbe8994c-cert\") pod \"ingress-canary-qrksf\" (UID: \"1ab9750b-0582-4c7c-b770-b1b7fbe8994c\") " pod="openshift-ingress-canary/ingress-canary-qrksf" Oct 30 06:56:23.288722 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:23.287684 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 06:56:23.288722 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:23.287770 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d930accb-e4d8-404b-8155-1b6a8edd812e-metrics-tls podName:d930accb-e4d8-404b-8155-1b6a8edd812e nodeName:}" failed. No retries permitted until 2026-10-30 06:56:23.7877461 +0000 UTC m=+30.916882605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d930accb-e4d8-404b-8155-1b6a8edd812e-metrics-tls") pod "dns-default-vgrqz" (UID: "d930accb-e4d8-404b-8155-1b6a8edd812e") : secret "dns-default-metrics-tls" not found Oct 30 06:56:23.288722 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.288426 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d930accb-e4d8-404b-8155-1b6a8edd812e-tmp-dir\") pod \"dns-default-vgrqz\" (UID: \"d930accb-e4d8-404b-8155-1b6a8edd812e\") " pod="openshift-dns/dns-default-vgrqz" Oct 30 06:56:23.288722 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.288626 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d930accb-e4d8-404b-8155-1b6a8edd812e-config-volume\") pod \"dns-default-vgrqz\" (UID: \"d930accb-e4d8-404b-8155-1b6a8edd812e\") " pod="openshift-dns/dns-default-vgrqz" Oct 30 06:56:23.291647 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.290499 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ab9750b-0582-4c7c-b770-b1b7fbe8994c-cert\") pod \"ingress-canary-qrksf\" (UID: \"1ab9750b-0582-4c7c-b770-b1b7fbe8994c\") " pod="openshift-ingress-canary/ingress-canary-qrksf" Oct 30 06:56:23.314864 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.314795 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgbhv\" (UniqueName: \"kubernetes.io/projected/d930accb-e4d8-404b-8155-1b6a8edd812e-kube-api-access-lgbhv\") pod \"dns-default-vgrqz\" (UID: \"d930accb-e4d8-404b-8155-1b6a8edd812e\") " pod="openshift-dns/dns-default-vgrqz" Oct 30 06:56:23.315871 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.315851 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fv48w" Oct 30 06:56:23.316168 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.316142 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx6zb\" (UniqueName: \"kubernetes.io/projected/1ab9750b-0582-4c7c-b770-b1b7fbe8994c-kube-api-access-jx6zb\") pod \"ingress-canary-qrksf\" (UID: \"1ab9750b-0582-4c7c-b770-b1b7fbe8994c\") " pod="openshift-ingress-canary/ingress-canary-qrksf" Oct 30 06:56:23.340077 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.340047 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2kt77" Oct 30 06:56:23.356079 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.354845 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z5nm2" Oct 30 06:56:23.373863 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.373799 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qrksf" Oct 30 06:56:23.462184 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.453447 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5bl7s"] Oct 30 06:56:23.462184 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.453750 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7dwjw"] Oct 30 06:56:23.462184 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:56:23.454104 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b28600_2ad3_47ff_a157_bcfacc6ab2ff.slice/crio-3d25172907c874087f97ffa70595329ed316947772248929150c3ac4dd0845e0 WatchSource:0}: Error finding container 3d25172907c874087f97ffa70595329ed316947772248929150c3ac4dd0845e0: Status 404 returned error can't find the container with id 3d25172907c874087f97ffa70595329ed316947772248929150c3ac4dd0845e0 Oct 30 06:56:23.462184 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:56:23.454742 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f93e69e_f48e_4c0a_ab3d_655739b6a8ae.slice/crio-7b8eac73c37aae02e42e006a6febb299eaa9e2812de39c0b1c0e87845301168f WatchSource:0}: Error finding container 7b8eac73c37aae02e42e006a6febb299eaa9e2812de39c0b1c0e87845301168f: Status 404 returned error can't find the container with id 7b8eac73c37aae02e42e006a6febb299eaa9e2812de39c0b1c0e87845301168f Oct 30 06:56:23.489527 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.488817 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5943f8dc-3e86-49fa-ac40-70a1f11fb69e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cjn9t\" (UID: \"5943f8dc-3e86-49fa-ac40-70a1f11fb69e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cjn9t" Oct 30 06:56:23.489527 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.488996 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l6qcv\" (UID: \"8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l6qcv" Oct 30 06:56:23.489527 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:23.489155 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 06:56:23.489527 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:23.489218 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-cluster-monitoring-operator-tls podName:8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:24.489199485 +0000 UTC m=+31.618335983 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-l6qcv" (UID: "8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2") : secret "cluster-monitoring-operator-tls" not found Oct 30 06:56:23.489527 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:23.489155 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 06:56:23.489527 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:23.489264 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5943f8dc-3e86-49fa-ac40-70a1f11fb69e-networking-console-plugin-cert podName:5943f8dc-3e86-49fa-ac40-70a1f11fb69e nodeName:}" failed. No retries permitted until 2026-10-30 06:56:24.489253838 +0000 UTC m=+31.618390338 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5943f8dc-3e86-49fa-ac40-70a1f11fb69e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-cjn9t" (UID: "5943f8dc-3e86-49fa-ac40-70a1f11fb69e") : secret "networking-console-plugin-cert" not found Oct 30 06:56:23.497098 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.496425 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zwfbr"] Oct 30 06:56:23.537945 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.537915 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-qqt8g"] Oct 30 06:56:23.545417 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:56:23.545378 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b59e668_d5a4_4ce0_a1e9_a16d63f51990.slice/crio-95e0179d73a99531da9cb392bfc0a488a6ec96b3ae881566ae7f46acd6766e53 WatchSource:0}: Error finding container 95e0179d73a99531da9cb392bfc0a488a6ec96b3ae881566ae7f46acd6766e53: Status 404 returned error can't find the container with id 95e0179d73a99531da9cb392bfc0a488a6ec96b3ae881566ae7f46acd6766e53 Oct 30 06:56:23.577586 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.577543 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-zw4km"] Oct 30 06:56:23.578643 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.578624 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8xtng"] Oct 30 06:56:23.590394 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.589764 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-service-ca-bundle\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:23.590394 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.589842 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-metrics-certs\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:23.590394 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.589921 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/50ab5bd7-ea08-460b-9932-1dac54f3005b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-cflrf\" (UID: \"50ab5bd7-ea08-460b-9932-1dac54f3005b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cflrf" Oct 30 06:56:23.590394 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:23.590148 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 06:56:23.590394 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:23.590204 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50ab5bd7-ea08-460b-9932-1dac54f3005b-samples-operator-tls podName:50ab5bd7-ea08-460b-9932-1dac54f3005b nodeName:}" failed. No retries permitted until 2026-10-30 06:56:24.590185545 +0000 UTC m=+31.719322039 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/50ab5bd7-ea08-460b-9932-1dac54f3005b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-cflrf" (UID: "50ab5bd7-ea08-460b-9932-1dac54f3005b") : secret "samples-operator-tls" not found Oct 30 06:56:23.590394 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:23.590277 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-service-ca-bundle podName:0c49a4a4-033f-4a62-8bc4-6b59531e8bf7 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:24.59026697 +0000 UTC m=+31.719403466 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-service-ca-bundle") pod "router-default-7f6bb8cb6f-fzpwq" (UID: "0c49a4a4-033f-4a62-8bc4-6b59531e8bf7") : configmap references non-existent config key: service-ca.crt Oct 30 06:56:23.590394 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:23.590334 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 06:56:23.590394 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:23.590364 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-metrics-certs podName:0c49a4a4-033f-4a62-8bc4-6b59531e8bf7 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:24.590354632 +0000 UTC m=+31.719491131 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-metrics-certs") pod "router-default-7f6bb8cb6f-fzpwq" (UID: "0c49a4a4-033f-4a62-8bc4-6b59531e8bf7") : secret "router-metrics-certs-default" not found Oct 30 06:56:23.632393 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.632205 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fv48w"] Oct 30 06:56:23.635779 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:56:23.635677 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35296121_39f4_4fae_ae2f_f92790ad3f5c.slice/crio-8ac1b866b9253e04ea641033e8e2583df1f52d0d1795eb4663b1407406e04a8c WatchSource:0}: Error finding container 8ac1b866b9253e04ea641033e8e2583df1f52d0d1795eb4663b1407406e04a8c: Status 404 returned error can't find the container with id 8ac1b866b9253e04ea641033e8e2583df1f52d0d1795eb4663b1407406e04a8c Oct 30 06:56:23.677529 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.677507 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-2kt77"] Oct 30 06:56:23.678760 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:56:23.678728 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5f1f83f_9bfd_45f5_b065_ba95cf659956.slice/crio-42a48dfd0e48d457bf0a01074309f6ebb34fd0abe78e8c5dc2ef2c854b221457 WatchSource:0}: Error finding container 42a48dfd0e48d457bf0a01074309f6ebb34fd0abe78e8c5dc2ef2c854b221457: Status 404 returned error can't find the container with id 42a48dfd0e48d457bf0a01074309f6ebb34fd0abe78e8c5dc2ef2c854b221457 Oct 30 06:56:23.688278 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.688242 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2kt77" event={"ID":"c5f1f83f-9bfd-45f5-b065-ba95cf659956","Type":"ContainerStarted","Data":"42a48dfd0e48d457bf0a01074309f6ebb34fd0abe78e8c5dc2ef2c854b221457"} Oct 30 06:56:23.689431 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.689405 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fv48w" event={"ID":"35296121-39f4-4fae-ae2f-f92790ad3f5c","Type":"ContainerStarted","Data":"8ac1b866b9253e04ea641033e8e2583df1f52d0d1795eb4663b1407406e04a8c"} Oct 30 06:56:23.691607 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.690588 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:23.691607 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.690630 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:23.691607 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.690632 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" event={"ID":"42802760-36a8-4b79-8690-5a00528503c7","Type":"ContainerStarted","Data":"9d6977b58f98e79134af3e36fac90409d296cb783b93165512a4aedc3442283f"} Oct 30 06:56:23.691607 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:23.690728 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 06:56:23.691607 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:23.690742 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b5f74d469-87gxs: secret "image-registry-tls" not found Oct 30 06:56:23.691607 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:23.690746 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 06:56:23.691607 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:23.690760 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dcb9bff4d-5xf9f: secret "image-registry-tls" not found Oct 30 06:56:23.691607 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:23.690784 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls podName:b9d1faf4-274f-49c8-9340-f85463840b27 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:24.690770976 +0000 UTC m=+31.819907482 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls") pod "image-registry-5b5f74d469-87gxs" (UID: "b9d1faf4-274f-49c8-9340-f85463840b27") : secret "image-registry-tls" not found Oct 30 06:56:23.691607 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:23.690797 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls podName:36b00a60-14fb-4364-8943-6bcacc5365cc nodeName:}" failed. No retries permitted until 2026-10-30 06:56:24.690791181 +0000 UTC m=+31.819927673 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls") pod "image-registry-5dcb9bff4d-5xf9f" (UID: "36b00a60-14fb-4364-8943-6bcacc5365cc") : secret "image-registry-tls" not found Oct 30 06:56:23.692208 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.692183 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-zwfbr" event={"ID":"239195cf-db90-4e49-9064-8f7aa4af9b3a","Type":"ContainerStarted","Data":"f5a34dd0a3c15a1392d3608321d2bb54b9e45b18b84a0ff3a7a644c216bdd584"} Oct 30 06:56:23.693439 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.693412 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-qqt8g" event={"ID":"4b59e668-d5a4-4ce0-a1e9-a16d63f51990","Type":"ContainerStarted","Data":"95e0179d73a99531da9cb392bfc0a488a6ec96b3ae881566ae7f46acd6766e53"} Oct 30 06:56:23.694990 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.694967 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5bl7s" event={"ID":"8f93e69e-f48e-4c0a-ab3d-655739b6a8ae","Type":"ContainerStarted","Data":"b73ec31c2ccab8b364d74dd5a5430af61d3178ea55bfb663bd712c876f4a9ffa"} Oct 30 06:56:23.695070 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.695002 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5bl7s" event={"ID":"8f93e69e-f48e-4c0a-ab3d-655739b6a8ae","Type":"ContainerStarted","Data":"7b8eac73c37aae02e42e006a6febb299eaa9e2812de39c0b1c0e87845301168f"} Oct 30 06:56:23.696467 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.696447 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7dwjw" event={"ID":"14b28600-2ad3-47ff-a157-bcfacc6ab2ff","Type":"ContainerStarted","Data":"81da9f59e50e73c4dd453df9b19793d65619ec6031bcd8079b15423de3e40333"} Oct 30 06:56:23.696592 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.696479 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7dwjw" event={"ID":"14b28600-2ad3-47ff-a157-bcfacc6ab2ff","Type":"ContainerStarted","Data":"3d25172907c874087f97ffa70595329ed316947772248929150c3ac4dd0845e0"} Oct 30 06:56:23.697674 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.697656 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8xtng" event={"ID":"20a9a026-75f0-4e90-b66a-066d3b80facf","Type":"ContainerStarted","Data":"b8602a9671a4e9c72f2110e3717c659f8ba28d538b2f1fb67fc540e9c952ba04"} Oct 30 06:56:23.733093 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.733071 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z5nm2"] Oct 30 06:56:23.734509 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:56:23.734478 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod584e4eb5_6e5e_4f96_b97a_eb58fd762347.slice/crio-0b31e106cc7fee0cc85211a463e39451f22c03f14d2a3dd1bc11229c82326b2b WatchSource:0}: Error finding container 0b31e106cc7fee0cc85211a463e39451f22c03f14d2a3dd1bc11229c82326b2b: Status 404 returned error can't find the container with id 0b31e106cc7fee0cc85211a463e39451f22c03f14d2a3dd1bc11229c82326b2b Oct 30 06:56:23.779739 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.779714 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qrksf"] Oct 30 06:56:23.781045 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:56:23.781019 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab9750b_0582_4c7c_b770_b1b7fbe8994c.slice/crio-2dc0083e51d76ca707b96c6a23ad632cb82d9d1c2cd39cc753e328e6dd78ff92 WatchSource:0}: Error finding container 2dc0083e51d76ca707b96c6a23ad632cb82d9d1c2cd39cc753e328e6dd78ff92: Status 404 returned error can't find the container with id 2dc0083e51d76ca707b96c6a23ad632cb82d9d1c2cd39cc753e328e6dd78ff92 Oct 30 06:56:23.791944 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:23.791919 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d930accb-e4d8-404b-8155-1b6a8edd812e-metrics-tls\") pod \"dns-default-vgrqz\" (UID: \"d930accb-e4d8-404b-8155-1b6a8edd812e\") " pod="openshift-dns/dns-default-vgrqz" Oct 30 06:56:23.792050 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:23.792033 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 06:56:23.792097 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:23.792088 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d930accb-e4d8-404b-8155-1b6a8edd812e-metrics-tls podName:d930accb-e4d8-404b-8155-1b6a8edd812e nodeName:}" failed. No retries permitted until 2026-10-30 06:56:24.79207357 +0000 UTC m=+31.921210063 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d930accb-e4d8-404b-8155-1b6a8edd812e-metrics-tls") pod "dns-default-vgrqz" (UID: "d930accb-e4d8-404b-8155-1b6a8edd812e") : secret "dns-default-metrics-tls" not found Oct 30 06:56:24.370040 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:24.369998 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:56:24.370040 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:24.370036 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:56:24.370290 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:24.369998 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjdmc" Oct 30 06:56:24.386002 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:24.385800 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 06:56:24.386002 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:24.385924 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-kkxlz\"" Oct 30 06:56:24.386402 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:24.386020 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-s89sh\"" Oct 30 06:56:24.386402 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:24.386190 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4sbng\"" Oct 30 06:56:24.386402 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:24.386218 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 06:56:24.501093 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:24.500208 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l6qcv\" (UID: \"8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l6qcv" Oct 30 06:56:24.501093 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:24.500314 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5943f8dc-3e86-49fa-ac40-70a1f11fb69e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cjn9t\" (UID: \"5943f8dc-3e86-49fa-ac40-70a1f11fb69e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cjn9t" Oct 30 06:56:24.501093 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:24.500480 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 06:56:24.501093 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:24.500542 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5943f8dc-3e86-49fa-ac40-70a1f11fb69e-networking-console-plugin-cert podName:5943f8dc-3e86-49fa-ac40-70a1f11fb69e nodeName:}" failed. No retries permitted until 2026-10-30 06:56:26.500523672 +0000 UTC m=+33.629660167 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5943f8dc-3e86-49fa-ac40-70a1f11fb69e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-cjn9t" (UID: "5943f8dc-3e86-49fa-ac40-70a1f11fb69e") : secret "networking-console-plugin-cert" not found Oct 30 06:56:24.501093 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:24.501003 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 06:56:24.501093 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:24.501060 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-cluster-monitoring-operator-tls podName:8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:26.501045224 +0000 UTC m=+33.630181722 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-l6qcv" (UID: "8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2") : secret "cluster-monitoring-operator-tls" not found Oct 30 06:56:24.602422 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:24.601156 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-service-ca-bundle\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:24.602422 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:24.601245 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-metrics-certs\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:24.602422 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:24.601318 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/50ab5bd7-ea08-460b-9932-1dac54f3005b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-cflrf\" (UID: \"50ab5bd7-ea08-460b-9932-1dac54f3005b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cflrf" Oct 30 06:56:24.602422 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:24.601883 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-service-ca-bundle podName:0c49a4a4-033f-4a62-8bc4-6b59531e8bf7 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:26.601865874 +0000 UTC m=+33.731002387 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-service-ca-bundle") pod "router-default-7f6bb8cb6f-fzpwq" (UID: "0c49a4a4-033f-4a62-8bc4-6b59531e8bf7") : configmap references non-existent config key: service-ca.crt Oct 30 06:56:24.602422 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:24.602219 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 06:56:24.602422 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:24.602270 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-metrics-certs podName:0c49a4a4-033f-4a62-8bc4-6b59531e8bf7 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:26.602253413 +0000 UTC m=+33.731389921 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-metrics-certs") pod "router-default-7f6bb8cb6f-fzpwq" (UID: "0c49a4a4-033f-4a62-8bc4-6b59531e8bf7") : secret "router-metrics-certs-default" not found Oct 30 06:56:24.602422 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:24.602314 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 06:56:24.602422 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:24.602349 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50ab5bd7-ea08-460b-9932-1dac54f3005b-samples-operator-tls podName:50ab5bd7-ea08-460b-9932-1dac54f3005b nodeName:}" failed. No retries permitted until 2026-10-30 06:56:26.60232863 +0000 UTC m=+33.731465138 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/50ab5bd7-ea08-460b-9932-1dac54f3005b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-cflrf" (UID: "50ab5bd7-ea08-460b-9932-1dac54f3005b") : secret "samples-operator-tls" not found Oct 30 06:56:24.702298 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:24.701852 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:24.702298 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:24.701912 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:24.702298 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:24.702156 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 06:56:24.702298 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:24.702172 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dcb9bff4d-5xf9f: secret "image-registry-tls" not found Oct 30 06:56:24.702298 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:24.702156 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 06:56:24.702298 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:24.702190 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b5f74d469-87gxs: secret "image-registry-tls" not found Oct 30 06:56:24.702298 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:24.702235 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls podName:36b00a60-14fb-4364-8943-6bcacc5365cc nodeName:}" failed. No retries permitted until 2026-10-30 06:56:26.702218533 +0000 UTC m=+33.831355029 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls") pod "image-registry-5dcb9bff4d-5xf9f" (UID: "36b00a60-14fb-4364-8943-6bcacc5365cc") : secret "image-registry-tls" not found Oct 30 06:56:24.702298 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:24.702250 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls podName:b9d1faf4-274f-49c8-9340-f85463840b27 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:26.702243167 +0000 UTC m=+33.831379661 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls") pod "image-registry-5b5f74d469-87gxs" (UID: "b9d1faf4-274f-49c8-9340-f85463840b27") : secret "image-registry-tls" not found Oct 30 06:56:24.707192 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:24.707152 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-zwfbr" event={"ID":"239195cf-db90-4e49-9064-8f7aa4af9b3a","Type":"ContainerStarted","Data":"ac92ee6fd3556cc682718388ba0c39256429254155fbf01aec458b5f0dbecc69"} Oct 30 06:56:24.709250 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:24.709219 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z5nm2" event={"ID":"584e4eb5-6e5e-4f96-b97a-eb58fd762347","Type":"ContainerStarted","Data":"0b31e106cc7fee0cc85211a463e39451f22c03f14d2a3dd1bc11229c82326b2b"} Oct 30 06:56:24.711078 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:24.711017 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qrksf" event={"ID":"1ab9750b-0582-4c7c-b770-b1b7fbe8994c","Type":"ContainerStarted","Data":"2dc0083e51d76ca707b96c6a23ad632cb82d9d1c2cd39cc753e328e6dd78ff92"} Oct 30 06:56:24.749246 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:24.747611 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-zwfbr" podStartSLOduration=384.747594884 podStartE2EDuration="6m24.747594884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:24.747172254 +0000 UTC m=+31.876308770" watchObservedRunningTime="2026-10-30 06:56:24.747594884 +0000 UTC m=+31.876731397" Oct 30 06:56:24.789605 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:24.789527 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-7dwjw" podStartSLOduration=384.789507396 podStartE2EDuration="6m24.789507396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:24.787772985 +0000 UTC m=+31.916909502" watchObservedRunningTime="2026-10-30 06:56:24.789507396 +0000 UTC m=+31.918643912" Oct 30 06:56:24.803919 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:24.803162 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d930accb-e4d8-404b-8155-1b6a8edd812e-metrics-tls\") pod \"dns-default-vgrqz\" (UID: \"d930accb-e4d8-404b-8155-1b6a8edd812e\") " pod="openshift-dns/dns-default-vgrqz" Oct 30 06:56:24.804089 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:24.803957 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 06:56:24.804089 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:24.804050 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d930accb-e4d8-404b-8155-1b6a8edd812e-metrics-tls podName:d930accb-e4d8-404b-8155-1b6a8edd812e nodeName:}" failed. No retries permitted until 2026-10-30 06:56:26.804031064 +0000 UTC m=+33.933167575 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d930accb-e4d8-404b-8155-1b6a8edd812e-metrics-tls") pod "dns-default-vgrqz" (UID: "d930accb-e4d8-404b-8155-1b6a8edd812e") : secret "dns-default-metrics-tls" not found Oct 30 06:56:24.827880 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:24.827821 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-5bl7s" podStartSLOduration=384.82780345 podStartE2EDuration="6m24.82780345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:24.826709825 +0000 UTC m=+31.955846340" watchObservedRunningTime="2026-10-30 06:56:24.82780345 +0000 UTC m=+31.956939964" Oct 30 06:56:26.521187 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:26.521146 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5943f8dc-3e86-49fa-ac40-70a1f11fb69e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cjn9t\" (UID: \"5943f8dc-3e86-49fa-ac40-70a1f11fb69e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cjn9t" Oct 30 06:56:26.521782 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:26.521306 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l6qcv\" (UID: \"8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l6qcv" Oct 30 06:56:26.521782 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:26.521321 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 06:56:26.521782 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:26.521402 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5943f8dc-3e86-49fa-ac40-70a1f11fb69e-networking-console-plugin-cert podName:5943f8dc-3e86-49fa-ac40-70a1f11fb69e nodeName:}" failed. No retries permitted until 2026-10-30 06:56:30.521378427 +0000 UTC m=+37.650514924 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5943f8dc-3e86-49fa-ac40-70a1f11fb69e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-cjn9t" (UID: "5943f8dc-3e86-49fa-ac40-70a1f11fb69e") : secret "networking-console-plugin-cert" not found Oct 30 06:56:26.521782 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:26.521406 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 06:56:26.521782 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:26.521447 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-cluster-monitoring-operator-tls podName:8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:30.52143606 +0000 UTC m=+37.650572553 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-l6qcv" (UID: "8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2") : secret "cluster-monitoring-operator-tls" not found Oct 30 06:56:26.622310 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:26.622265 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/50ab5bd7-ea08-460b-9932-1dac54f3005b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-cflrf\" (UID: \"50ab5bd7-ea08-460b-9932-1dac54f3005b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cflrf" Oct 30 06:56:26.622509 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:26.622429 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-service-ca-bundle\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:26.622509 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:26.622497 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-metrics-certs\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:26.622713 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:26.622651 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 06:56:26.622775 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:26.622717 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-metrics-certs podName:0c49a4a4-033f-4a62-8bc4-6b59531e8bf7 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:30.62269924 +0000 UTC m=+37.751835750 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-metrics-certs") pod "router-default-7f6bb8cb6f-fzpwq" (UID: "0c49a4a4-033f-4a62-8bc4-6b59531e8bf7") : secret "router-metrics-certs-default" not found Oct 30 06:56:26.622841 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:26.622790 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 06:56:26.622841 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:26.622822 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50ab5bd7-ea08-460b-9932-1dac54f3005b-samples-operator-tls podName:50ab5bd7-ea08-460b-9932-1dac54f3005b nodeName:}" failed. No retries permitted until 2026-10-30 06:56:30.622810543 +0000 UTC m=+37.751947049 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/50ab5bd7-ea08-460b-9932-1dac54f3005b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-cflrf" (UID: "50ab5bd7-ea08-460b-9932-1dac54f3005b") : secret "samples-operator-tls" not found Oct 30 06:56:26.622946 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:26.622887 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-service-ca-bundle podName:0c49a4a4-033f-4a62-8bc4-6b59531e8bf7 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:30.62287721 +0000 UTC m=+37.752013719 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-service-ca-bundle") pod "router-default-7f6bb8cb6f-fzpwq" (UID: "0c49a4a4-033f-4a62-8bc4-6b59531e8bf7") : configmap references non-existent config key: service-ca.crt Oct 30 06:56:26.723608 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:26.722870 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:26.723608 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:26.722918 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:26.723608 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:26.723155 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 06:56:26.723608 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:26.723167 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b5f74d469-87gxs: secret "image-registry-tls" not found Oct 30 06:56:26.723608 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:26.723206 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls podName:b9d1faf4-274f-49c8-9340-f85463840b27 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:30.723194467 +0000 UTC m=+37.852330960 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls") pod "image-registry-5b5f74d469-87gxs" (UID: "b9d1faf4-274f-49c8-9340-f85463840b27") : secret "image-registry-tls" not found Oct 30 06:56:26.723608 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:26.723279 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 06:56:26.723608 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:26.723289 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dcb9bff4d-5xf9f: secret "image-registry-tls" not found Oct 30 06:56:26.723608 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:26.723318 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls podName:36b00a60-14fb-4364-8943-6bcacc5365cc nodeName:}" failed. No retries permitted until 2026-10-30 06:56:30.723307941 +0000 UTC m=+37.852444448 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls") pod "image-registry-5dcb9bff4d-5xf9f" (UID: "36b00a60-14fb-4364-8943-6bcacc5365cc") : secret "image-registry-tls" not found Oct 30 06:56:26.725795 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:26.725102 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z5nm2" event={"ID":"584e4eb5-6e5e-4f96-b97a-eb58fd762347","Type":"ContainerStarted","Data":"d38f5ff63d1755e29f811b00ee2cb838963764533ba2ae3a18b7a95f3e2dfb77"} Oct 30 06:56:26.766782 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:26.766723 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z5nm2" podStartSLOduration=351.829050182 podStartE2EDuration="5m53.766703973s" totalImagesPullingTime="1.937653791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:50:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:26.76602464 +0000 UTC m=+33.895161156" watchObservedRunningTime="2026-10-30 06:56:26.766703973 +0000 UTC m=+33.895840490" Oct 30 06:56:26.824718 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:26.823984 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d930accb-e4d8-404b-8155-1b6a8edd812e-metrics-tls\") pod \"dns-default-vgrqz\" (UID: \"d930accb-e4d8-404b-8155-1b6a8edd812e\") " pod="openshift-dns/dns-default-vgrqz" Oct 30 06:56:26.824718 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:26.824227 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 06:56:26.824718 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:26.824293 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d930accb-e4d8-404b-8155-1b6a8edd812e-metrics-tls podName:d930accb-e4d8-404b-8155-1b6a8edd812e nodeName:}" failed. No retries permitted until 2026-10-30 06:56:30.824274132 +0000 UTC m=+37.953410644 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d930accb-e4d8-404b-8155-1b6a8edd812e-metrics-tls") pod "dns-default-vgrqz" (UID: "d930accb-e4d8-404b-8155-1b6a8edd812e") : secret "dns-default-metrics-tls" not found Oct 30 06:56:27.128186 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:27.127179 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-metrics-certs\") pod \"network-metrics-daemon-g6vbv\" (UID: \"f1b14112-6c97-4bfa-8f2a-5988f59ad06a\") " pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:56:27.128186 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:27.127338 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-original-pull-secret\") pod \"global-pull-secret-syncer-ts77g\" (UID: \"96024683-dd4e-4d78-b5ad-6aa9ccc4bc05\") " pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:56:27.128186 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:27.127604 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 06:56:27.128186 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:27.127661 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-metrics-certs podName:f1b14112-6c97-4bfa-8f2a-5988f59ad06a nodeName:}" failed. No retries permitted until 2026-10-30 06:56:59.127641997 +0000 UTC m=+66.256778491 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-metrics-certs") pod "network-metrics-daemon-g6vbv" (UID: "f1b14112-6c97-4bfa-8f2a-5988f59ad06a") : secret "metrics-daemon-secret" not found Oct 30 06:56:27.131024 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:27.130997 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-original-pull-secret\") pod \"global-pull-secret-syncer-ts77g\" (UID: \"96024683-dd4e-4d78-b5ad-6aa9ccc4bc05\") " pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:56:27.228533 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:27.228489 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7ndt\" (UniqueName: \"kubernetes.io/projected/16e1a28b-a156-4d4e-820f-8712616b4eb2-kube-api-access-x7ndt\") pod \"network-check-target-rjdmc\" (UID: \"16e1a28b-a156-4d4e-820f-8712616b4eb2\") " pod="openshift-network-diagnostics/network-check-target-rjdmc" Oct 30 06:56:27.233625 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:27.233567 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7ndt\" (UniqueName: \"kubernetes.io/projected/16e1a28b-a156-4d4e-820f-8712616b4eb2-kube-api-access-x7ndt\") pod \"network-check-target-rjdmc\" (UID: \"16e1a28b-a156-4d4e-820f-8712616b4eb2\") " pod="openshift-network-diagnostics/network-check-target-rjdmc" Oct 30 06:56:27.403742 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:27.403658 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:56:27.459448 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:27.459226 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjdmc" Oct 30 06:56:27.730276 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:27.730205 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ts77g"] Oct 30 06:56:27.792518 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:56:27.792463 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96024683_dd4e_4d78_b5ad_6aa9ccc4bc05.slice/crio-75c0d71e22dcc3d3037d884ef030bdfd57b2dafb30ec4aeacc323020d0917b53 WatchSource:0}: Error finding container 75c0d71e22dcc3d3037d884ef030bdfd57b2dafb30ec4aeacc323020d0917b53: Status 404 returned error can't find the container with id 75c0d71e22dcc3d3037d884ef030bdfd57b2dafb30ec4aeacc323020d0917b53 Oct 30 06:56:27.814411 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:27.814238 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rjdmc"] Oct 30 06:56:28.732399 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:28.732332 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ts77g" event={"ID":"96024683-dd4e-4d78-b5ad-6aa9ccc4bc05","Type":"ContainerStarted","Data":"75c0d71e22dcc3d3037d884ef030bdfd57b2dafb30ec4aeacc323020d0917b53"} Oct 30 06:56:28.733679 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:28.733648 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rjdmc" event={"ID":"16e1a28b-a156-4d4e-820f-8712616b4eb2","Type":"ContainerStarted","Data":"2b91b75787a17708af821dc397864d622c6d6d570031d443fd6341c5121dab71"} Oct 30 06:56:29.741059 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:29.740347 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8xtng" event={"ID":"20a9a026-75f0-4e90-b66a-066d3b80facf","Type":"ContainerStarted","Data":"561f1d8ed87b1ca5db56adba97783fd27d143734c74e9bb0263e95f1f45620bd"} Oct 30 06:56:29.742233 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:29.742202 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qrksf" event={"ID":"1ab9750b-0582-4c7c-b770-b1b7fbe8994c","Type":"ContainerStarted","Data":"c0f06e81c4b6ed65ecbd3cad2ea71fbe4468bfac30b583d28d5db7c883860675"} Oct 30 06:56:29.751103 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:29.751078 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2kt77" event={"ID":"c5f1f83f-9bfd-45f5-b065-ba95cf659956","Type":"ContainerStarted","Data":"81f3c61a54268340de3e4cc760b6ba988c88d5bc62a23656f86998d0205394cd"} Oct 30 06:56:29.753488 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:29.753461 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fv48w" event={"ID":"35296121-39f4-4fae-ae2f-f92790ad3f5c","Type":"ContainerStarted","Data":"88b5183983e8afc28665c76c340f3ae5c771b457d911b38b61bce0c3cf553a86"} Oct 30 06:56:29.755397 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:29.755377 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/0.log" Oct 30 06:56:29.755502 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:29.755419 2574 generic.go:356] "Generic (PLEG): container finished" podID="42802760-36a8-4b79-8690-5a00528503c7" containerID="101394f6e419dc884cb0c45f8ed7e0e556e85d13439c55a1dcd492daace26f72" exitCode=255 Oct 30 06:56:29.755619 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:29.755599 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" event={"ID":"42802760-36a8-4b79-8690-5a00528503c7","Type":"ContainerDied","Data":"101394f6e419dc884cb0c45f8ed7e0e556e85d13439c55a1dcd492daace26f72"} Oct 30 06:56:29.755694 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:29.755666 2574 scope.go:122] "RemoveContainer" containerID="101394f6e419dc884cb0c45f8ed7e0e556e85d13439c55a1dcd492daace26f72" Oct 30 06:56:29.759339 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:29.759317 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-qqt8g" event={"ID":"4b59e668-d5a4-4ce0-a1e9-a16d63f51990","Type":"ContainerStarted","Data":"71efa7e0684157232e61eaaf212a9c38bb649bb585571d2d04f9bbb2e3cec4d3"} Oct 30 06:56:29.765600 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:29.765022 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rjdmc" event={"ID":"16e1a28b-a156-4d4e-820f-8712616b4eb2","Type":"ContainerStarted","Data":"4dad733ad899b70b9176889a345e419957ee852f878cc05f83a4c04ef584c70f"} Oct 30 06:56:29.765600 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:29.765155 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rjdmc" Oct 30 06:56:29.788663 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:29.788622 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8xtng" podStartSLOduration=351.64795893 podStartE2EDuration="5m56.7886101s" totalImagesPullingTime="5.14065117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:50:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:29.787889014 +0000 UTC m=+36.917025531" watchObservedRunningTime="2026-10-30 06:56:29.7886101 +0000 UTC m=+36.917746615" Oct 30 06:56:29.882767 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:29.882663 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qrksf" podStartSLOduration=2.363216039 podStartE2EDuration="7.882644981s" totalImagesPullingTime="5.519428942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:29.880773934 +0000 UTC m=+37.009910450" watchObservedRunningTime="2026-10-30 06:56:29.882644981 +0000 UTC m=+37.011781501" Oct 30 06:56:29.884351 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:29.883242 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-qqt8g" podStartSLOduration=351.709830978 podStartE2EDuration="5m56.883234542s" totalImagesPullingTime="5.173403564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:50:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:29.836068464 +0000 UTC m=+36.965204978" watchObservedRunningTime="2026-10-30 06:56:29.883234542 +0000 UTC m=+37.012371058" Oct 30 06:56:29.965348 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:29.965315 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-r5bpp"] Oct 30 06:56:29.972061 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:29.972005 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2kt77" podStartSLOduration=351.929968621 podStartE2EDuration="5m56.971987418s" totalImagesPullingTime="5.042018797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:50:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:29.969753791 +0000 UTC m=+37.098890307" watchObservedRunningTime="2026-10-30 06:56:29.971987418 +0000 UTC m=+37.101123932" Oct 30 06:56:29.988992 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:29.988926 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:30.009358 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.009216 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r5bpp"] Oct 30 06:56:30.039717 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.039526 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fv48w" podStartSLOduration=351.389028358 podStartE2EDuration="5m57.039510607s" totalImagesPullingTime="5.650482249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:50:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:30.036015838 +0000 UTC m=+37.165152356" watchObservedRunningTime="2026-10-30 06:56:30.039510607 +0000 UTC m=+37.168647123" Oct 30 06:56:30.063526 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.063495 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-data-volume\") pod \"insights-runtime-extractor-r5bpp\" (UID: \"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3\") " pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:30.063706 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.063534 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r5bpp\" (UID: \"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3\") " pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:30.063706 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.063564 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tf48\" (UniqueName: \"kubernetes.io/projected/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-kube-api-access-2tf48\") pod \"insights-runtime-extractor-r5bpp\" (UID: \"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3\") " pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:30.063826 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.063735 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r5bpp\" (UID: \"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3\") " pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:30.063895 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.063873 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-crio-socket\") pod \"insights-runtime-extractor-r5bpp\" (UID: \"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3\") " pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:30.080320 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.080275 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rjdmc" podStartSLOduration=35.361902623 podStartE2EDuration="37.080260088s" totalImagesPullingTime="1.718357465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:55:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:30.079203312 +0000 UTC m=+37.208339827" watchObservedRunningTime="2026-10-30 06:56:30.080260088 +0000 UTC m=+37.209396604" Oct 30 06:56:30.091455 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.091429 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 06:56:30.091619 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.091470 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7mg7c\"" Oct 30 06:56:30.112829 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.112797 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 06:56:30.165154 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.165074 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-data-volume\") pod \"insights-runtime-extractor-r5bpp\" (UID: \"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3\") " pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:30.165154 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.165113 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r5bpp\" (UID: \"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3\") " pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:30.165354 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:30.165187 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 06:56:30.165354 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:30.165233 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-kube-rbac-proxy-cm podName:4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:30.665219417 +0000 UTC m=+37.794355910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r5bpp" (UID: "4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3") : configmap "kube-rbac-proxy" not found Oct 30 06:56:30.165354 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.165250 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tf48\" (UniqueName: \"kubernetes.io/projected/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-kube-api-access-2tf48\") pod \"insights-runtime-extractor-r5bpp\" (UID: \"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3\") " pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:30.165354 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.165305 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r5bpp\" (UID: \"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3\") " pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:30.165550 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.165394 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-crio-socket\") pod \"insights-runtime-extractor-r5bpp\" (UID: \"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3\") " pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:30.165550 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.165426 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-data-volume\") pod \"insights-runtime-extractor-r5bpp\" (UID: \"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3\") " pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:30.165550 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.165509 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-crio-socket\") pod \"insights-runtime-extractor-r5bpp\" (UID: \"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3\") " pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:30.165710 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:30.165550 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 06:56:30.165710 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:30.165638 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-insights-runtime-extractor-tls podName:4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:30.665610981 +0000 UTC m=+37.794747491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r5bpp" (UID: "4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3") : secret "insights-runtime-extractor-tls" not found Oct 30 06:56:30.191781 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.191748 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tf48\" (UniqueName: \"kubernetes.io/projected/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-kube-api-access-2tf48\") pod \"insights-runtime-extractor-r5bpp\" (UID: \"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3\") " pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:30.342996 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.342965 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6t2h6"] Oct 30 06:56:30.359718 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.359685 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6t2h6" Oct 30 06:56:30.369109 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.369079 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6t2h6"] Oct 30 06:56:30.467882 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.467759 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jldj6\" (UniqueName: \"kubernetes.io/projected/1327b481-6353-4d2d-8847-caa7fc7e3a3d-kube-api-access-jldj6\") pod \"migrator-66df5d496b-6t2h6\" (UID: \"1327b481-6353-4d2d-8847-caa7fc7e3a3d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6t2h6" Oct 30 06:56:30.467882 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.467766 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 06:56:30.467882 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.467798 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 06:56:30.468182 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.468007 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-7qqzj\"" Oct 30 06:56:30.568867 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.568821 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jldj6\" (UniqueName: \"kubernetes.io/projected/1327b481-6353-4d2d-8847-caa7fc7e3a3d-kube-api-access-jldj6\") pod \"migrator-66df5d496b-6t2h6\" (UID: \"1327b481-6353-4d2d-8847-caa7fc7e3a3d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6t2h6" Oct 30 06:56:30.569054 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.569030 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l6qcv\" (UID: \"8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l6qcv" Oct 30 06:56:30.569131 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.569106 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5943f8dc-3e86-49fa-ac40-70a1f11fb69e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cjn9t\" (UID: \"5943f8dc-3e86-49fa-ac40-70a1f11fb69e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cjn9t" Oct 30 06:56:30.569225 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:30.569194 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 06:56:30.569287 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:30.569223 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 06:56:30.569338 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:30.569289 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5943f8dc-3e86-49fa-ac40-70a1f11fb69e-networking-console-plugin-cert podName:5943f8dc-3e86-49fa-ac40-70a1f11fb69e nodeName:}" failed. No retries permitted until 2026-10-30 06:56:38.569269935 +0000 UTC m=+45.698406454 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5943f8dc-3e86-49fa-ac40-70a1f11fb69e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-cjn9t" (UID: "5943f8dc-3e86-49fa-ac40-70a1f11fb69e") : secret "networking-console-plugin-cert" not found Oct 30 06:56:30.569338 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:30.569308 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-cluster-monitoring-operator-tls podName:8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:38.569299286 +0000 UTC m=+45.698435783 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-l6qcv" (UID: "8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2") : secret "cluster-monitoring-operator-tls" not found Oct 30 06:56:30.615614 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.615589 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jldj6\" (UniqueName: \"kubernetes.io/projected/1327b481-6353-4d2d-8847-caa7fc7e3a3d-kube-api-access-jldj6\") pod \"migrator-66df5d496b-6t2h6\" (UID: \"1327b481-6353-4d2d-8847-caa7fc7e3a3d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6t2h6" Oct 30 06:56:30.669988 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.669956 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r5bpp\" (UID: \"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3\") " pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:30.670187 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.670005 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/50ab5bd7-ea08-460b-9932-1dac54f3005b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-cflrf\" (UID: \"50ab5bd7-ea08-460b-9932-1dac54f3005b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cflrf" Oct 30 06:56:30.670187 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:30.670107 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 06:56:30.670187 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.670124 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-service-ca-bundle\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:30.670187 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:30.670156 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 06:56:30.670187 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:30.670175 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-insights-runtime-extractor-tls podName:4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:31.67015705 +0000 UTC m=+38.799293566 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r5bpp" (UID: "4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3") : secret "insights-runtime-extractor-tls" not found Oct 30 06:56:30.670375 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:30.670200 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50ab5bd7-ea08-460b-9932-1dac54f3005b-samples-operator-tls podName:50ab5bd7-ea08-460b-9932-1dac54f3005b nodeName:}" failed. No retries permitted until 2026-10-30 06:56:38.670190887 +0000 UTC m=+45.799327383 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/50ab5bd7-ea08-460b-9932-1dac54f3005b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-cflrf" (UID: "50ab5bd7-ea08-460b-9932-1dac54f3005b") : secret "samples-operator-tls" not found Oct 30 06:56:30.670375 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.670247 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r5bpp\" (UID: \"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3\") " pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:30.670375 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:30.670259 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-service-ca-bundle podName:0c49a4a4-033f-4a62-8bc4-6b59531e8bf7 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:38.670242342 +0000 UTC m=+45.799378850 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-service-ca-bundle") pod "router-default-7f6bb8cb6f-fzpwq" (UID: "0c49a4a4-033f-4a62-8bc4-6b59531e8bf7") : configmap references non-existent config key: service-ca.crt Oct 30 06:56:30.670375 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:30.670295 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 06:56:30.670375 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.670295 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-metrics-certs\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:30.670375 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:30.670327 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-kube-rbac-proxy-cm podName:4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:31.67031757 +0000 UTC m=+38.799454096 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r5bpp" (UID: "4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3") : configmap "kube-rbac-proxy" not found Oct 30 06:56:30.670375 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:30.670366 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 06:56:30.670604 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:30.670398 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-metrics-certs podName:0c49a4a4-033f-4a62-8bc4-6b59531e8bf7 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:38.670389872 +0000 UTC m=+45.799526400 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-metrics-certs") pod "router-default-7f6bb8cb6f-fzpwq" (UID: "0c49a4a4-033f-4a62-8bc4-6b59531e8bf7") : secret "router-metrics-certs-default" not found Oct 30 06:56:30.688138 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.688110 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6t2h6" Oct 30 06:56:30.771097 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.771068 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:30.771485 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.771113 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:30.771485 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:30.771341 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 06:56:30.771485 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:30.771353 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b5f74d469-87gxs: secret "image-registry-tls" not found Oct 30 06:56:30.771485 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:30.771394 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls podName:b9d1faf4-274f-49c8-9340-f85463840b27 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:38.771379886 +0000 UTC m=+45.900516378 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls") pod "image-registry-5b5f74d469-87gxs" (UID: "b9d1faf4-274f-49c8-9340-f85463840b27") : secret "image-registry-tls" not found Oct 30 06:56:30.771703 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:30.771687 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 06:56:30.771703 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:30.771697 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dcb9bff4d-5xf9f: secret "image-registry-tls" not found Oct 30 06:56:30.771800 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:30.771731 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls podName:36b00a60-14fb-4364-8943-6bcacc5365cc nodeName:}" failed. No retries permitted until 2026-10-30 06:56:38.771720556 +0000 UTC m=+45.900857049 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls") pod "image-registry-5dcb9bff4d-5xf9f" (UID: "36b00a60-14fb-4364-8943-6bcacc5365cc") : secret "image-registry-tls" not found Oct 30 06:56:30.771800 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.770502 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/1.log" Oct 30 06:56:30.772308 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.772285 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/0.log" Oct 30 06:56:30.772388 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.772325 2574 generic.go:356] "Generic (PLEG): container finished" podID="42802760-36a8-4b79-8690-5a00528503c7" containerID="ab6a3aae9beab6299cb2ae4ffc208236cde7fa979fe642aba021008f4c9e948f" exitCode=255 Oct 30 06:56:30.775055 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.773226 2574 scope.go:122] "RemoveContainer" containerID="ab6a3aae9beab6299cb2ae4ffc208236cde7fa979fe642aba021008f4c9e948f" Oct 30 06:56:30.775055 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:30.773396 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zw4km_openshift-console-operator(42802760-36a8-4b79-8690-5a00528503c7)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" podUID="42802760-36a8-4b79-8690-5a00528503c7" Oct 30 06:56:30.775055 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.773444 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" event={"ID":"42802760-36a8-4b79-8690-5a00528503c7","Type":"ContainerDied","Data":"ab6a3aae9beab6299cb2ae4ffc208236cde7fa979fe642aba021008f4c9e948f"} Oct 30 06:56:30.775055 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.773490 2574 scope.go:122] "RemoveContainer" containerID="101394f6e419dc884cb0c45f8ed7e0e556e85d13439c55a1dcd492daace26f72" Oct 30 06:56:30.873145 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:30.873116 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d930accb-e4d8-404b-8155-1b6a8edd812e-metrics-tls\") pod \"dns-default-vgrqz\" (UID: \"d930accb-e4d8-404b-8155-1b6a8edd812e\") " pod="openshift-dns/dns-default-vgrqz" Oct 30 06:56:30.873479 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:30.873462 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 06:56:30.873608 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:30.873507 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d930accb-e4d8-404b-8155-1b6a8edd812e-metrics-tls podName:d930accb-e4d8-404b-8155-1b6a8edd812e nodeName:}" failed. No retries permitted until 2026-10-30 06:56:38.873493295 +0000 UTC m=+46.002629788 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d930accb-e4d8-404b-8155-1b6a8edd812e-metrics-tls") pod "dns-default-vgrqz" (UID: "d930accb-e4d8-404b-8155-1b6a8edd812e") : secret "dns-default-metrics-tls" not found Oct 30 06:56:31.000285 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:31.000256 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6t2h6"] Oct 30 06:56:31.004731 ip-10-0-132-173 kubenswrapper[2574]: W1030 06:56:31.004694 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1327b481_6353_4d2d_8847_caa7fc7e3a3d.slice/crio-b55fd4cb6b13ceba4c7bb3248705871e89a027d215528df2338601bab68c0470 WatchSource:0}: Error finding container b55fd4cb6b13ceba4c7bb3248705871e89a027d215528df2338601bab68c0470: Status 404 returned error can't find the container with id b55fd4cb6b13ceba4c7bb3248705871e89a027d215528df2338601bab68c0470 Oct 30 06:56:31.681027 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:31.680992 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r5bpp\" (UID: \"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3\") " pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:31.681202 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:31.681146 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r5bpp\" (UID: \"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3\") " pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:31.681202 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:31.681150 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 06:56:31.681285 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:31.681219 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-insights-runtime-extractor-tls podName:4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:33.681203417 +0000 UTC m=+40.810339936 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r5bpp" (UID: "4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3") : secret "insights-runtime-extractor-tls" not found Oct 30 06:56:31.681285 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:31.681255 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 06:56:31.681367 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:31.681293 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-kube-rbac-proxy-cm podName:4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:33.681282677 +0000 UTC m=+40.810419188 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r5bpp" (UID: "4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3") : configmap "kube-rbac-proxy" not found Oct 30 06:56:31.778134 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:31.778094 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6t2h6" event={"ID":"1327b481-6353-4d2d-8847-caa7fc7e3a3d","Type":"ContainerStarted","Data":"b55fd4cb6b13ceba4c7bb3248705871e89a027d215528df2338601bab68c0470"} Oct 30 06:56:31.779991 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:31.779966 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/1.log" Oct 30 06:56:31.780369 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:31.780350 2574 scope.go:122] "RemoveContainer" containerID="ab6a3aae9beab6299cb2ae4ffc208236cde7fa979fe642aba021008f4c9e948f" Oct 30 06:56:31.780639 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:31.780611 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zw4km_openshift-console-operator(42802760-36a8-4b79-8690-5a00528503c7)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" podUID="42802760-36a8-4b79-8690-5a00528503c7" Oct 30 06:56:32.785284 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:32.785249 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6t2h6" event={"ID":"1327b481-6353-4d2d-8847-caa7fc7e3a3d","Type":"ContainerStarted","Data":"312ae3c23ef62580d7e27cae19a9ad1dbbfef506d0ccf991ecb077edaf288ebb"} Oct 30 06:56:32.827126 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:32.827065 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6t2h6" podStartSLOduration=1.344105664 podStartE2EDuration="2.827048933s" totalImagesPullingTime="1.482943269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:56:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:32.825955779 +0000 UTC m=+39.955092295" watchObservedRunningTime="2026-10-30 06:56:32.827048933 +0000 UTC m=+39.956185497" Oct 30 06:56:33.234108 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:33.234032 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" Oct 30 06:56:33.234237 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:33.234117 2574 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" Oct 30 06:56:33.234461 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:33.234447 2574 scope.go:122] "RemoveContainer" containerID="ab6a3aae9beab6299cb2ae4ffc208236cde7fa979fe642aba021008f4c9e948f" Oct 30 06:56:33.234672 ip-10-0-132-173 kubenswrapper[2574]: E1030 06:56:33.234652 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zw4km_openshift-console-operator(42802760-36a8-4b79-8690-5a00528503c7)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" podUID="42802760-36a8-4b79-8690-5a00528503c7" Oct 30 06:56:33.563931 ip-10-0-132-173 kubenswrapper[2574]: I1030 06:56:33.563792 2574 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 06:56:33.563937 ip-10-0-132-173 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 06:56:33.581085 ip-10-0-132-173 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 06:56:33.581314 ip-10-0-132-173 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 06:56:33.581380 ip-10-0-132-173 systemd[1]: kubelet.service: Consumed 2.997s CPU time. Oct 30 06:56:33.595156 ip-10-0-132-173 systemd[1]: Starting Kubernetes Kubelet... Oct 30 06:56:33.685894 ip-10-0-132-173 kubenswrapper[4951]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 06:56:33.685894 ip-10-0-132-173 kubenswrapper[4951]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 06:56:33.685894 ip-10-0-132-173 kubenswrapper[4951]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 06:56:33.685894 ip-10-0-132-173 kubenswrapper[4951]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 06:56:33.688130 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688049 4951 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:56:33.688130 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688124 4951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:56:33.688130 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688129 4951 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:56:33.688130 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688133 4951 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:56:33.688130 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688137 4951 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:56:33.688456 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688142 4951 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:56:33.688456 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688147 4951 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:56:33.688456 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688151 4951 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:56:33.688456 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688154 4951 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:56:33.688456 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688158 4951 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:56:33.688456 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688162 4951 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:56:33.688456 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688165 4951 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:56:33.688456 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688169 4951 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:56:33.688456 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688173 4951 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:56:33.688456 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688176 4951 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:56:33.688456 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688181 4951 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:56:33.688456 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688184 4951 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:56:33.688456 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688188 4951 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:56:33.688456 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688192 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:56:33.688456 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688197 4951 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:56:33.688456 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688201 4951 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:56:33.688456 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688206 4951 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:56:33.688456 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688211 4951 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:56:33.688456 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688215 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:56:33.688456 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688218 4951 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:56:33.689340 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688222 4951 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:56:33.689340 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688226 4951 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:56:33.689340 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688239 4951 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:56:33.689340 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688243 4951 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:56:33.689340 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688247 4951 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:56:33.689340 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688251 4951 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:56:33.689340 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688255 4951 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:56:33.689340 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688259 4951 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:56:33.689340 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688263 4951 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:56:33.689340 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688267 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:56:33.689340 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688271 4951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:56:33.689340 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688276 4951 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:56:33.689340 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688280 4951 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:56:33.689340 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688284 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:56:33.689340 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688289 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:56:33.689340 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688293 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:56:33.689340 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688297 4951 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:56:33.689340 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688305 4951 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:56:33.689340 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688312 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:56:33.690194 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688317 4951 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:56:33.690194 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688321 4951 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:56:33.690194 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688326 4951 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:56:33.690194 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688331 4951 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:56:33.690194 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688335 4951 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:56:33.690194 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688340 4951 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:56:33.690194 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688344 4951 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:56:33.690194 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688348 4951 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:56:33.690194 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688353 4951 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:56:33.690194 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688357 4951 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:56:33.690194 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688361 4951 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:56:33.690194 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688367 4951 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:56:33.690194 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688371 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:56:33.690194 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688375 4951 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:56:33.690194 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688380 4951 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:56:33.690194 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688384 4951 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:56:33.690194 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688388 4951 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:56:33.690194 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688392 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:56:33.690194 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688397 4951 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:56:33.690194 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688402 4951 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:56:33.691081 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688407 4951 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:56:33.691081 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688411 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:56:33.691081 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688415 4951 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:56:33.691081 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688421 4951 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:56:33.691081 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688425 4951 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:56:33.691081 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688430 4951 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:56:33.691081 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688434 4951 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:56:33.691081 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688438 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:56:33.691081 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688442 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:56:33.691081 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688447 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:56:33.691081 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688451 4951 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:56:33.691081 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688455 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:56:33.691081 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688459 4951 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:56:33.691081 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688463 4951 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:56:33.691081 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688468 4951 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:56:33.691081 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688473 4951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:56:33.691081 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688477 4951 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:56:33.691081 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688481 4951 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:56:33.691081 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688486 4951 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:56:33.692003 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688490 4951 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:56:33.692003 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688494 4951 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:56:33.692003 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688499 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:56:33.692003 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688505 4951 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:56:33.692003 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688512 4951 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:56:33.692003 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688517 4951 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:56:33.692003 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688521 4951 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:56:33.692003 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688526 4951 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:56:33.692003 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688530 4951 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:56:33.692003 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688535 4951 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:56:33.692003 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688539 4951 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:56:33.692003 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688543 4951 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:56:33.692003 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688548 4951 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:56:33.692003 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688553 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:56:33.692003 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688557 4951 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:56:33.692003 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688561 4951 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:56:33.692003 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688566 4951 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:56:33.692003 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688586 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:56:33.692003 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688590 4951 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:56:33.692880 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688593 4951 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:56:33.692880 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688597 4951 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:56:33.692880 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688601 4951 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:56:33.692880 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.688605 4951 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:56:33.692880 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.689896 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:56:33.692880 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.689906 4951 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:56:33.692880 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.689910 4951 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:56:33.692880 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.689915 4951 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:56:33.692880 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.689919 4951 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:56:33.692880 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.689924 4951 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:56:33.692880 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.689929 4951 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:56:33.692880 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.689933 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:56:33.692880 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.689937 4951 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:56:33.692880 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.689942 4951 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:56:33.692880 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.689946 4951 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:56:33.692880 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.689951 4951 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:56:33.692880 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.689955 4951 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:56:33.692880 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.689959 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:56:33.692880 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.689964 4951 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:56:33.693774 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.689968 4951 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:56:33.693774 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.689972 4951 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:56:33.693774 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.689977 4951 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:56:33.693774 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.689982 4951 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:56:33.693774 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.689986 4951 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:56:33.693774 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.689990 4951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:56:33.693774 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.689995 4951 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:56:33.693774 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690001 4951 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:56:33.693774 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690005 4951 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:56:33.693774 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690010 4951 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:56:33.693774 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690014 4951 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:56:33.693774 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690018 4951 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:56:33.693774 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690023 4951 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:56:33.693774 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690029 4951 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:56:33.693774 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690035 4951 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:56:33.693774 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690040 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:56:33.693774 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690047 4951 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:56:33.693774 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690054 4951 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:56:33.693774 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690059 4951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:56:33.694515 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690063 4951 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:56:33.694515 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690068 4951 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:56:33.694515 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690072 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:56:33.694515 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690076 4951 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:56:33.694515 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690081 4951 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:56:33.694515 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690086 4951 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:56:33.694515 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690090 4951 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:56:33.694515 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690094 4951 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:56:33.694515 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690098 4951 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:56:33.694515 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690102 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:56:33.694515 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690106 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:56:33.694515 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690111 4951 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:56:33.694515 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690115 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:56:33.694515 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690120 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:56:33.694515 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690124 4951 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:56:33.694515 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690128 4951 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:56:33.694515 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690133 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:56:33.694515 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690137 4951 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:56:33.694515 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690143 4951 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:56:33.694515 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690147 4951 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:56:33.695033 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690151 4951 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:56:33.695033 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690156 4951 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:56:33.695033 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690160 4951 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:56:33.695033 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690164 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:56:33.695033 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690169 4951 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:56:33.695033 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690173 4951 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:56:33.695033 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690177 4951 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:56:33.695033 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690181 4951 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:56:33.695033 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690185 4951 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:56:33.695033 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690189 4951 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:56:33.695033 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690194 4951 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:56:33.695033 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690198 4951 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:56:33.695033 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690203 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:56:33.695033 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690207 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:56:33.695033 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690212 4951 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:56:33.695033 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690216 4951 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:56:33.695033 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690220 4951 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:56:33.695033 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690225 4951 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:56:33.695033 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690230 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:56:33.695493 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690234 4951 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:56:33.695493 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690238 4951 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:56:33.695493 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690242 4951 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:56:33.695493 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690247 4951 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:56:33.695493 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690252 4951 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:56:33.695493 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690256 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:56:33.695493 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690261 4951 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:56:33.695493 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690265 4951 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:56:33.695493 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690270 4951 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:56:33.695493 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690274 4951 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:56:33.695493 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690279 4951 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:56:33.695493 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690283 4951 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:56:33.695493 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690287 4951 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:56:33.695493 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690292 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:56:33.695493 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690296 4951 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:56:33.695493 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690300 4951 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:56:33.695493 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690304 4951 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:56:33.695493 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690308 4951 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:56:33.695493 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690313 4951 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:56:33.695493 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690317 4951 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:56:33.696040 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690322 4951 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:56:33.696040 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690326 4951 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:56:33.696040 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690330 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:56:33.696040 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690334 4951 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:56:33.696040 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690338 4951 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:56:33.696040 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690347 4951 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:56:33.696040 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690351 4951 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:56:33.696040 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690355 4951 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:56:33.696040 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690360 4951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:56:33.696040 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690364 4951 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:56:33.696040 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690368 4951 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:56:33.696040 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690372 4951 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:56:33.696040 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.690380 4951 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:56:33.696040 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691194 4951 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:56:33.696040 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691205 4951 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:56:33.696040 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691209 4951 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:56:33.696040 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691213 4951 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:56:33.696040 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691217 4951 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:56:33.696040 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691222 4951 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:56:33.696040 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691226 4951 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:56:33.696525 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691231 4951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:56:33.696525 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691235 4951 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:56:33.696525 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691242 4951 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:56:33.696525 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691248 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:56:33.696525 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691253 4951 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:56:33.696525 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691257 4951 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:56:33.696525 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691262 4951 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:56:33.696525 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691266 4951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:56:33.696525 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691270 4951 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:56:33.696525 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691275 4951 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:56:33.696525 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691279 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:56:33.696525 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691284 4951 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:56:33.696525 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691288 4951 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:56:33.696525 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691292 4951 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:56:33.696525 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691296 4951 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:56:33.696525 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691300 4951 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:56:33.696525 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691306 4951 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:56:33.696525 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691313 4951 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:56:33.696525 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691317 4951 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:56:33.697016 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691321 4951 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:56:33.697016 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691326 4951 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:56:33.697016 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691330 4951 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:56:33.697016 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691335 4951 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:56:33.697016 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691339 4951 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:56:33.697016 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691345 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:56:33.697016 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691349 4951 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:56:33.697016 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691353 4951 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:56:33.697016 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691358 4951 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:56:33.697016 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691362 4951 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:56:33.697016 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691367 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:56:33.697016 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691371 4951 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:56:33.697016 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691375 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:56:33.697016 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691380 4951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:56:33.697016 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691384 4951 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:56:33.697016 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691388 4951 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:56:33.697016 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691392 4951 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:56:33.697016 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691396 4951 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:56:33.697016 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691400 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:56:33.697016 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691405 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:56:33.697507 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691409 4951 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:56:33.697507 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691413 4951 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:56:33.697507 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691417 4951 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:56:33.697507 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691422 4951 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:56:33.697507 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691426 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:56:33.697507 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691431 4951 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:56:33.697507 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691435 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:56:33.697507 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691439 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:56:33.697507 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691443 4951 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:56:33.697507 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691448 4951 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:56:33.697507 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691456 4951 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:56:33.697507 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691460 4951 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:56:33.697507 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691465 4951 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:56:33.697507 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691470 4951 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:56:33.697507 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691475 4951 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:56:33.697507 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691479 4951 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:56:33.697507 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691484 4951 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:56:33.697507 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691488 4951 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:56:33.697507 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691493 4951 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:56:33.697507 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691497 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:56:33.698158 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691501 4951 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:56:33.698158 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691506 4951 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:56:33.698158 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691510 4951 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:56:33.698158 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691514 4951 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:56:33.698158 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691518 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:56:33.698158 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691522 4951 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:56:33.698158 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691526 4951 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:56:33.698158 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691530 4951 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:56:33.698158 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691534 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:56:33.698158 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691540 4951 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:56:33.698158 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691546 4951 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:56:33.698158 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691550 4951 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:56:33.698158 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691555 4951 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:56:33.698158 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691560 4951 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:56:33.698158 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691565 4951 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:56:33.698158 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691587 4951 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:56:33.698158 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691591 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:56:33.698158 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691595 4951 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:56:33.698158 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691600 4951 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:56:33.698913 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691604 4951 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:56:33.698913 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691609 4951 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:56:33.698913 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691613 4951 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:56:33.698913 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691619 4951 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:56:33.698913 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691623 4951 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:56:33.698913 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691627 4951 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:56:33.698913 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691632 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:56:33.698913 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691636 4951 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:56:33.698913 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691642 4951 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:56:33.698913 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691647 4951 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:56:33.698913 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691652 4951 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:56:33.698913 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691656 4951 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:56:33.698913 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691661 4951 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:56:33.698913 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691665 4951 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:56:33.698913 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691669 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:56:33.698913 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691673 4951 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:56:33.698913 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691678 4951 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:56:33.698913 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691682 4951 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:56:33.698913 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691686 4951 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:56:33.698913 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691690 4951 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:56:33.699509 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.691695 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:56:33.699509 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.691831 4951 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 06:56:33.699509 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.691842 4951 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 06:56:33.699509 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.691851 4951 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 06:56:33.699509 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.691858 4951 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 06:56:33.699509 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.691865 4951 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 06:56:33.699509 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.691870 4951 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 06:56:33.699509 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.691877 4951 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 06:56:33.699509 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.691883 4951 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 06:56:33.699509 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.691889 4951 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 06:56:33.699509 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.691894 4951 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 06:56:33.699509 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.691900 4951 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 06:56:33.699509 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.691906 4951 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 06:56:33.699509 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.691911 4951 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 06:56:33.699509 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.691915 4951 flags.go:64] FLAG: --cgroup-root="" Oct 30 06:56:33.699509 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.691922 4951 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 06:56:33.699509 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.691927 4951 flags.go:64] FLAG: --client-ca-file="" Oct 30 06:56:33.699509 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.691932 4951 flags.go:64] FLAG: --cloud-provider="external" Oct 30 06:56:33.699509 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.691937 4951 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 06:56:33.699509 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.691944 4951 flags.go:64] FLAG: --cluster-domain="" Oct 30 06:56:33.699509 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.691949 4951 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 06:56:33.699509 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.691954 4951 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 06:56:33.699509 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.691959 4951 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 06:56:33.699509 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.691966 4951 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 06:56:33.700103 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.691972 4951 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 06:56:33.700103 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.691978 4951 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 06:56:33.700103 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.691983 4951 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 06:56:33.700103 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.691988 4951 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 06:56:33.700103 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.691993 4951 flags.go:64] FLAG: --contention-profiling="false" Oct 30 06:56:33.700103 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.691998 4951 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 06:56:33.700103 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692003 4951 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 06:56:33.700103 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692007 4951 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 06:56:33.700103 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692012 4951 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 06:56:33.700103 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692019 4951 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 06:56:33.700103 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692023 4951 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 06:56:33.700103 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692028 4951 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 06:56:33.700103 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692033 4951 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 06:56:33.700103 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692038 4951 flags.go:64] FLAG: --enable-server="true" Oct 30 06:56:33.700103 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692042 4951 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 06:56:33.700103 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692049 4951 flags.go:64] FLAG: --event-burst="100" Oct 30 06:56:33.700103 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692054 4951 flags.go:64] FLAG: --event-qps="50" Oct 30 06:56:33.700103 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692059 4951 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 06:56:33.700103 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692063 4951 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 06:56:33.700103 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692071 4951 flags.go:64] FLAG: --eviction-hard="" Oct 30 06:56:33.700103 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692078 4951 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 06:56:33.700103 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692083 4951 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 06:56:33.700103 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692088 4951 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 06:56:33.700103 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692095 4951 flags.go:64] FLAG: --eviction-soft="" Oct 30 06:56:33.700103 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692100 4951 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 06:56:33.700763 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692104 4951 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 06:56:33.700763 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692109 4951 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 06:56:33.700763 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692114 4951 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 06:56:33.700763 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692119 4951 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 06:56:33.700763 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692124 4951 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 06:56:33.700763 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692128 4951 flags.go:64] FLAG: --feature-gates="" Oct 30 06:56:33.700763 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692135 4951 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 06:56:33.700763 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692140 4951 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 06:56:33.700763 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692145 4951 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 06:56:33.700763 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692151 4951 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 06:56:33.700763 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692156 4951 flags.go:64] FLAG: --healthz-port="10248" Oct 30 06:56:33.700763 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692161 4951 flags.go:64] FLAG: --help="false" Oct 30 06:56:33.700763 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692166 4951 flags.go:64] FLAG: --hostname-override="ip-10-0-132-173.ec2.internal" Oct 30 06:56:33.700763 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692171 4951 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 06:56:33.700763 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692176 4951 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 06:56:33.700763 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692181 4951 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 06:56:33.700763 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692186 4951 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 06:56:33.700763 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692192 4951 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 06:56:33.700763 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692196 4951 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 06:56:33.700763 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692201 4951 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 06:56:33.700763 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692205 4951 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 06:56:33.700763 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692210 4951 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 06:56:33.700763 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692215 4951 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 06:56:33.701321 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692221 4951 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 06:56:33.701321 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692225 4951 flags.go:64] FLAG: --kube-reserved="" Oct 30 06:56:33.701321 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692230 4951 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 06:56:33.701321 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692237 4951 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 06:56:33.701321 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692242 4951 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 06:56:33.701321 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692246 4951 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 06:56:33.701321 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692251 4951 flags.go:64] FLAG: --lock-file="" Oct 30 06:56:33.701321 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692258 4951 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 06:56:33.701321 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692263 4951 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 06:56:33.701321 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692268 4951 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 06:56:33.701321 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692277 4951 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 06:56:33.701321 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692282 4951 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 06:56:33.701321 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692287 4951 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 06:56:33.701321 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692291 4951 flags.go:64] FLAG: --logging-format="text" Oct 30 06:56:33.701321 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692297 4951 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 06:56:33.701321 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692303 4951 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 06:56:33.701321 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692307 4951 flags.go:64] FLAG: --manifest-url="" Oct 30 06:56:33.701321 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692312 4951 flags.go:64] FLAG: --manifest-url-header="" Oct 30 06:56:33.701321 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692319 4951 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 06:56:33.701321 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692324 4951 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 06:56:33.701321 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692330 4951 flags.go:64] FLAG: --max-pods="110" Oct 30 06:56:33.701321 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692335 4951 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 06:56:33.701321 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692340 4951 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 06:56:33.701321 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692344 4951 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 06:56:33.701321 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692349 4951 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 06:56:33.701955 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692354 4951 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 06:56:33.701955 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692359 4951 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 06:56:33.701955 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692364 4951 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 06:56:33.701955 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692374 4951 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 06:56:33.701955 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692379 4951 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 06:56:33.701955 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692384 4951 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 06:56:33.701955 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692389 4951 flags.go:64] FLAG: --pod-cidr="" Oct 30 06:56:33.701955 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692394 4951 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 06:56:33.701955 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692399 4951 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 06:56:33.701955 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692404 4951 flags.go:64] FLAG: --pods-per-core="0" Oct 30 06:56:33.701955 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692409 4951 flags.go:64] FLAG: --port="10250" Oct 30 06:56:33.701955 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692414 4951 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 06:56:33.701955 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692419 4951 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-088610d53ff64ae41" Oct 30 06:56:33.701955 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692424 4951 flags.go:64] FLAG: --qos-reserved="" Oct 30 06:56:33.701955 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692430 4951 flags.go:64] FLAG: --read-only-port="10255" Oct 30 06:56:33.701955 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692436 4951 flags.go:64] FLAG: --register-node="true" Oct 30 06:56:33.701955 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692440 4951 flags.go:64] FLAG: --register-with-taints="" Oct 30 06:56:33.701955 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692446 4951 flags.go:64] FLAG: --registry-burst="10" Oct 30 06:56:33.701955 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692451 4951 flags.go:64] FLAG: --registry-qps="5" Oct 30 06:56:33.701955 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692456 4951 flags.go:64] FLAG: --reserved-cpus="" Oct 30 06:56:33.701955 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692460 4951 flags.go:64] FLAG: --reserved-memory="" Oct 30 06:56:33.701955 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692466 4951 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 06:56:33.701955 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692472 4951 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 06:56:33.701955 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692476 4951 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 06:56:33.701955 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692481 4951 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 06:56:33.701955 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692486 4951 flags.go:64] FLAG: --runonce="false" Oct 30 06:56:33.702584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692491 4951 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 06:56:33.702584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692497 4951 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 06:56:33.702584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692502 4951 flags.go:64] FLAG: --seccomp-default="false" Oct 30 06:56:33.702584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692506 4951 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 06:56:33.702584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692512 4951 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 06:56:33.702584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692517 4951 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 06:56:33.702584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692521 4951 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 06:56:33.702584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692526 4951 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 06:56:33.702584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692531 4951 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 06:56:33.702584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692536 4951 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 06:56:33.702584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692540 4951 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 06:56:33.702584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692545 4951 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 06:56:33.702584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692550 4951 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 06:56:33.702584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692555 4951 flags.go:64] FLAG: --system-cgroups="" Oct 30 06:56:33.702584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692559 4951 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 06:56:33.702584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692585 4951 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 06:56:33.702584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692592 4951 flags.go:64] FLAG: --tls-cert-file="" Oct 30 06:56:33.702584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692597 4951 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 06:56:33.702584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692604 4951 flags.go:64] FLAG: --tls-min-version="" Oct 30 06:56:33.702584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692609 4951 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 06:56:33.702584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692615 4951 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 06:56:33.702584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692620 4951 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 06:56:33.702584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692625 4951 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 06:56:33.702584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692629 4951 flags.go:64] FLAG: --v="2" Oct 30 06:56:33.702584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692636 4951 flags.go:64] FLAG: --version="false" Oct 30 06:56:33.703275 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692643 4951 flags.go:64] FLAG: --vmodule="" Oct 30 06:56:33.703275 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692649 4951 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 06:56:33.703275 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.692655 4951 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 06:56:33.703275 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.692870 4951 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:56:33.703275 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.692877 4951 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:56:33.703275 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.692881 4951 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:56:33.703275 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.692886 4951 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:56:33.703275 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.692890 4951 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:56:33.703275 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.692894 4951 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:56:33.703275 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.692899 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:56:33.703275 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.692903 4951 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:56:33.703275 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.692907 4951 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:56:33.703275 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.692912 4951 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:56:33.703275 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.692916 4951 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:56:33.703275 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.692921 4951 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:56:33.703275 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.692925 4951 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:56:33.703275 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.692930 4951 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:56:33.703275 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.692934 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:56:33.703275 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.692938 4951 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:56:33.703275 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.692942 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:56:33.703874 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.692947 4951 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:56:33.703874 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.692951 4951 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:56:33.703874 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.692955 4951 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:56:33.703874 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.692962 4951 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:56:33.703874 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.692966 4951 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:56:33.703874 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.692971 4951 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:56:33.703874 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.692983 4951 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:56:33.703874 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.692990 4951 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:56:33.703874 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.692994 4951 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:56:33.703874 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.692999 4951 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:56:33.703874 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693003 4951 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:56:33.703874 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693008 4951 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:56:33.703874 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693012 4951 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:56:33.703874 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693019 4951 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:56:33.703874 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693026 4951 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:56:33.703874 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693031 4951 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:56:33.703874 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693037 4951 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:56:33.703874 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693042 4951 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:56:33.703874 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693047 4951 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:56:33.704334 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693052 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:56:33.704334 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693056 4951 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:56:33.704334 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693060 4951 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:56:33.704334 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693066 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:56:33.704334 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693071 4951 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:56:33.704334 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693075 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:56:33.704334 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693079 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:56:33.704334 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693084 4951 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:56:33.704334 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693088 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:56:33.704334 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693093 4951 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:56:33.704334 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693097 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:56:33.704334 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693101 4951 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:56:33.704334 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693112 4951 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:56:33.704334 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693116 4951 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:56:33.704334 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693120 4951 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:56:33.704334 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693125 4951 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:56:33.704334 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693131 4951 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:56:33.704334 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693135 4951 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:56:33.704334 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693140 4951 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:56:33.704852 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693144 4951 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:56:33.704852 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693150 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:56:33.704852 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693154 4951 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:56:33.704852 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693159 4951 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:56:33.704852 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693163 4951 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:56:33.704852 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693167 4951 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:56:33.704852 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693172 4951 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:56:33.704852 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693176 4951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:56:33.704852 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693180 4951 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:56:33.704852 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693184 4951 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:56:33.704852 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693189 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:56:33.704852 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693193 4951 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:56:33.704852 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693198 4951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:56:33.704852 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693202 4951 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:56:33.704852 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693207 4951 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:56:33.704852 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693211 4951 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:56:33.704852 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693215 4951 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:56:33.704852 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693220 4951 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:56:33.704852 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693224 4951 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:56:33.704852 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693228 4951 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:56:33.705333 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693233 4951 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:56:33.705333 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693237 4951 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:56:33.705333 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693241 4951 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:56:33.705333 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693246 4951 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:56:33.705333 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693250 4951 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:56:33.705333 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693254 4951 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:56:33.705333 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693258 4951 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:56:33.705333 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693263 4951 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:56:33.705333 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693267 4951 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:56:33.705333 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693273 4951 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:56:33.705333 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693278 4951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:56:33.705333 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693283 4951 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:56:33.705333 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693287 4951 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:56:33.705333 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693293 4951 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:56:33.705333 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693297 4951 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:56:33.705333 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693301 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:56:33.705333 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693306 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:56:33.705333 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693310 4951 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:56:33.705333 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693314 4951 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:56:33.706028 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693319 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:56:33.706028 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693323 4951 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:56:33.706028 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693329 4951 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:56:33.706028 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693335 4951 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:56:33.706028 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693339 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:56:33.706028 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693344 4951 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:56:33.706028 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693349 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:56:33.706028 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693353 4951 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:56:33.706028 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693357 4951 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:56:33.706028 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693362 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:56:33.706028 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693366 4951 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:56:33.706028 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693371 4951 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:56:33.706028 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.693412 4951 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 06:56:33.706028 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693471 4951 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:56:33.706028 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693478 4951 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:56:33.706435 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693482 4951 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:56:33.706435 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693486 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:56:33.706435 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693491 4951 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:56:33.706435 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693495 4951 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:56:33.706435 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693499 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:56:33.706435 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693504 4951 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:56:33.706435 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693508 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:56:33.706435 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693514 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:56:33.706435 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693519 4951 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:56:33.706435 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693523 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:56:33.706435 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693528 4951 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:56:33.706435 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693533 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:56:33.706435 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693538 4951 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:56:33.706435 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693543 4951 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:56:33.706435 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693547 4951 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:56:33.706435 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693551 4951 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:56:33.706435 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693555 4951 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:56:33.706435 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693560 4951 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:56:33.706435 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693564 4951 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:56:33.706990 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693583 4951 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:56:33.706990 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693588 4951 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:56:33.706990 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693592 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:56:33.706990 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693597 4951 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:56:33.706990 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693601 4951 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:56:33.706990 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693605 4951 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:56:33.706990 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693610 4951 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:56:33.706990 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693614 4951 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:56:33.706990 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693618 4951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:56:33.706990 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693622 4951 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:56:33.706990 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693628 4951 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:56:33.706990 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693632 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:56:33.706990 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693636 4951 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:56:33.706990 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693641 4951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:56:33.706990 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693645 4951 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:56:33.706990 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693650 4951 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:56:33.706990 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693654 4951 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:56:33.706990 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693658 4951 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:56:33.706990 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693663 4951 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:56:33.706990 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693667 4951 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:56:33.707523 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693673 4951 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:56:33.707523 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693677 4951 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:56:33.707523 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693682 4951 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:56:33.707523 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693686 4951 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:56:33.707523 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693690 4951 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:56:33.707523 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693697 4951 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:56:33.707523 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693701 4951 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:56:33.707523 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693705 4951 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:56:33.707523 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693709 4951 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:56:33.707523 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693714 4951 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:56:33.707523 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693718 4951 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:56:33.707523 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693723 4951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:56:33.707523 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693727 4951 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:56:33.707523 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693731 4951 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:56:33.707523 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693735 4951 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:56:33.707523 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693740 4951 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:56:33.707523 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693744 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:56:33.707523 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693748 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:56:33.707523 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693753 4951 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:56:33.708008 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693757 4951 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:56:33.708008 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693762 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:56:33.708008 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693766 4951 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:56:33.708008 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693771 4951 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:56:33.708008 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693778 4951 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:56:33.708008 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693783 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:56:33.708008 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693789 4951 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:56:33.708008 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693794 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:56:33.708008 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693799 4951 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:56:33.708008 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693803 4951 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:56:33.708008 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693807 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:56:33.708008 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693812 4951 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:56:33.708008 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693816 4951 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:56:33.708008 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693823 4951 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:56:33.708008 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693827 4951 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:56:33.708008 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693832 4951 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:56:33.708008 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693836 4951 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:56:33.708008 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693840 4951 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:56:33.708008 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693845 4951 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:56:33.708458 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693849 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:56:33.708458 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693853 4951 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:56:33.708458 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693857 4951 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:56:33.708458 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693862 4951 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:56:33.708458 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693866 4951 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:56:33.708458 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693870 4951 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:56:33.708458 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693874 4951 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:56:33.708458 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693879 4951 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:56:33.708458 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693883 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:56:33.708458 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693887 4951 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:56:33.708458 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693891 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:56:33.708458 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693896 4951 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:56:33.708458 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693900 4951 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:56:33.708458 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693905 4951 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:56:33.708458 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693909 4951 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:56:33.708458 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693913 4951 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:56:33.708458 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693917 4951 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:56:33.708458 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693921 4951 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:56:33.708458 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693926 4951 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:56:33.708458 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693931 4951 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:56:33.708953 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693935 4951 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:56:33.708953 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693939 4951 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:56:33.708953 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693943 4951 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:56:33.708953 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693947 4951 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:56:33.708953 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693954 4951 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:56:33.708953 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693961 4951 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:56:33.708953 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.693967 4951 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:56:33.708953 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.697567 4951 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 06:56:33.708953 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.697595 4951 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 06:56:33.708953 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.697614 4951 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 06:56:33.708953 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.697618 4951 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 06:56:33.708953 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697671 4951 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:56:33.708953 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697676 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:56:33.708953 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697679 4951 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:56:33.708953 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697682 4951 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:56:33.708953 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697686 4951 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:56:33.708953 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697688 4951 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:56:33.708953 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697691 4951 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:56:33.708953 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697694 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:56:33.709428 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697697 4951 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:56:33.709428 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697699 4951 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:56:33.709428 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697702 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:56:33.709428 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697705 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:56:33.709428 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697707 4951 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:56:33.709428 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697710 4951 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:56:33.709428 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697713 4951 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:56:33.709428 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697715 4951 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:56:33.709428 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697719 4951 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:56:33.709428 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697723 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:56:33.709428 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697726 4951 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:56:33.709428 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697729 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:56:33.709428 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697732 4951 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:56:33.709428 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697736 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:56:33.709428 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697739 4951 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:56:33.709428 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697742 4951 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:56:33.709428 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697745 4951 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:56:33.709428 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697748 4951 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:56:33.709428 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697751 4951 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:56:33.709428 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697753 4951 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:56:33.710012 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697756 4951 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:56:33.710012 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697758 4951 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:56:33.710012 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697761 4951 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:56:33.710012 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697764 4951 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:56:33.710012 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697766 4951 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:56:33.710012 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697769 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:56:33.710012 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697771 4951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:56:33.710012 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697774 4951 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:56:33.710012 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697777 4951 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:56:33.710012 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697779 4951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:56:33.710012 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697782 4951 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:56:33.710012 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697784 4951 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:56:33.710012 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697787 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:56:33.710012 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697789 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:56:33.710012 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697792 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:56:33.710012 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697795 4951 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:56:33.710012 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697797 4951 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:56:33.710012 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697800 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:56:33.710012 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697802 4951 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:56:33.710484 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697805 4951 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:56:33.710484 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697807 4951 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:56:33.710484 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697810 4951 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:56:33.710484 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697813 4951 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:56:33.710484 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697815 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:56:33.710484 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697818 4951 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:56:33.710484 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697821 4951 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:56:33.710484 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697823 4951 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:56:33.710484 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697827 4951 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:56:33.710484 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697830 4951 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:56:33.710484 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697833 4951 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:56:33.710484 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697835 4951 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:56:33.710484 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697838 4951 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:56:33.710484 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697840 4951 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:56:33.710484 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697843 4951 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:56:33.710484 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697846 4951 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:56:33.710484 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697848 4951 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:56:33.710484 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697850 4951 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:56:33.710484 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697853 4951 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:56:33.710951 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697856 4951 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:56:33.710951 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697858 4951 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:56:33.710951 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697861 4951 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:56:33.710951 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697864 4951 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:56:33.710951 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697866 4951 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:56:33.710951 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697869 4951 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:56:33.710951 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697871 4951 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:56:33.710951 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697874 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:56:33.710951 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697877 4951 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:56:33.710951 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697879 4951 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:56:33.710951 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697882 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:56:33.710951 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697884 4951 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:56:33.710951 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697887 4951 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:56:33.710951 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697889 4951 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:56:33.710951 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697892 4951 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:56:33.710951 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697895 4951 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:56:33.710951 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697897 4951 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:56:33.710951 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697900 4951 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:56:33.710951 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697902 4951 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:56:33.710951 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697905 4951 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:56:33.711414 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697907 4951 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:56:33.711414 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697910 4951 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:56:33.711414 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697914 4951 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:56:33.711414 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697917 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:56:33.711414 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697921 4951 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:56:33.711414 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697923 4951 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:56:33.711414 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697926 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:56:33.711414 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697929 4951 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:56:33.711414 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697931 4951 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:56:33.711414 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697934 4951 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:56:33.711414 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697937 4951 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:56:33.711414 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697940 4951 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:56:33.711414 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697943 4951 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:56:33.711414 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697945 4951 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:56:33.711414 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697948 4951 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:56:33.711414 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697950 4951 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:56:33.711414 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697953 4951 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:56:33.711414 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697955 4951 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:56:33.711414 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697958 4951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:56:33.711923 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.697960 4951 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:56:33.711923 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.697987 4951 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 06:56:33.711923 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.698855 4951 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:56:33.711923 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.698869 4951 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:56:33.711923 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.698872 4951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:56:33.711923 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.698875 4951 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:56:33.711923 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.698878 4951 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:56:33.711923 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.698881 4951 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:56:33.711923 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.698884 4951 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:56:33.711923 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.698969 4951 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:56:33.711923 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.698986 4951 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:56:33.711923 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.698991 4951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:56:33.711923 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.698996 4951 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:56:33.711923 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699001 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:56:33.711923 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699005 4951 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:56:33.711923 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699010 4951 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:56:33.712300 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699015 4951 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:56:33.712300 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699019 4951 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:56:33.712300 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699025 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:56:33.712300 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699030 4951 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:56:33.712300 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699034 4951 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:56:33.712300 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699045 4951 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:56:33.712300 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699049 4951 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:56:33.712300 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699052 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:56:33.712300 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699057 4951 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:56:33.712300 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699061 4951 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:56:33.712300 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699065 4951 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:56:33.712300 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699069 4951 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:56:33.712300 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699073 4951 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:56:33.712300 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699077 4951 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:56:33.712300 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699082 4951 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:56:33.712300 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699087 4951 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:56:33.712300 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699091 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:56:33.712300 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699126 4951 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:56:33.712300 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699138 4951 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:56:33.712760 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699143 4951 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:56:33.712760 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699147 4951 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:56:33.712760 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699152 4951 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:56:33.712760 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699156 4951 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:56:33.712760 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699160 4951 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:56:33.712760 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699209 4951 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:56:33.712760 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699325 4951 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:56:33.712760 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699329 4951 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:56:33.712760 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699332 4951 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:56:33.712760 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699336 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:56:33.712760 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699339 4951 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:56:33.712760 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699342 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:56:33.712760 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699345 4951 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:56:33.712760 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699347 4951 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:56:33.712760 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699350 4951 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:56:33.712760 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699353 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:56:33.712760 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699355 4951 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:56:33.712760 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699358 4951 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:56:33.712760 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699361 4951 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:56:33.713221 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699363 4951 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:56:33.713221 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699366 4951 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:56:33.713221 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699369 4951 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:56:33.713221 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699372 4951 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:56:33.713221 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699374 4951 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:56:33.713221 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699380 4951 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:56:33.713221 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699382 4951 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:56:33.713221 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699385 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:56:33.713221 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699388 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:56:33.713221 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699390 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:56:33.713221 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699393 4951 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:56:33.713221 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699395 4951 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:56:33.713221 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699398 4951 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:56:33.713221 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699402 4951 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:56:33.713221 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699405 4951 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:56:33.713221 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699408 4951 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:56:33.713221 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699411 4951 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:56:33.713221 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699414 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:56:33.713221 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699416 4951 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:56:33.713890 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699419 4951 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:56:33.713890 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699422 4951 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:56:33.713890 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699425 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:56:33.713890 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699428 4951 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:56:33.713890 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699430 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:56:33.713890 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699433 4951 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:56:33.713890 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699436 4951 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:56:33.713890 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699438 4951 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:56:33.713890 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699450 4951 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:56:33.713890 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699454 4951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:56:33.713890 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699457 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:56:33.713890 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699460 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:56:33.713890 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699465 4951 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:56:33.713890 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699467 4951 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:56:33.713890 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699470 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:56:33.713890 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699473 4951 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:56:33.713890 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699475 4951 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:56:33.713890 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699478 4951 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:56:33.713890 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699482 4951 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:56:33.713890 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699485 4951 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:56:33.714692 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699487 4951 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:56:33.714692 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699490 4951 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:56:33.714692 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699492 4951 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:56:33.714692 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699497 4951 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:56:33.714692 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699501 4951 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:56:33.714692 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699504 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:56:33.714692 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699507 4951 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:56:33.714692 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699510 4951 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:56:33.714692 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699513 4951 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:56:33.714692 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699516 4951 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:56:33.714692 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699519 4951 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:56:33.714692 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699521 4951 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:56:33.714692 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699524 4951 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:56:33.714692 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699527 4951 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:56:33.714692 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699530 4951 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:56:33.714692 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.699567 4951 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 06:56:33.715246 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699631 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:56:33.715246 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699636 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:56:33.715246 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699639 4951 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:56:33.715246 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699642 4951 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:56:33.715246 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699645 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:56:33.715246 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699647 4951 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:56:33.715246 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699650 4951 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:56:33.715246 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699652 4951 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:56:33.715246 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699655 4951 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:56:33.715246 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699658 4951 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:56:33.715246 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699661 4951 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:56:33.715246 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699663 4951 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:56:33.715246 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699666 4951 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:56:33.715246 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699670 4951 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:56:33.715246 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699674 4951 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:56:33.715246 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699680 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:56:33.715246 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699684 4951 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:56:33.715246 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699687 4951 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:56:33.715246 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699690 4951 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:56:33.715246 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699692 4951 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:56:33.715747 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699695 4951 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:56:33.715747 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699698 4951 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:56:33.715747 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699700 4951 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:56:33.715747 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699703 4951 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:56:33.715747 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699705 4951 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:56:33.715747 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699709 4951 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:56:33.715747 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699711 4951 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:56:33.715747 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699714 4951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:56:33.715747 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699717 4951 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:56:33.715747 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699719 4951 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:56:33.715747 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699722 4951 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:56:33.715747 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699724 4951 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:56:33.715747 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699727 4951 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:56:33.715747 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699730 4951 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:56:33.715747 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699732 4951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:56:33.715747 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699735 4951 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:56:33.715747 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699737 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:56:33.715747 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699740 4951 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:56:33.715747 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699743 4951 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:56:33.716195 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699745 4951 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:56:33.716195 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699748 4951 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:56:33.716195 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699751 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:56:33.716195 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699760 4951 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:56:33.716195 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699763 4951 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:56:33.716195 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699766 4951 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:56:33.716195 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699768 4951 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:56:33.716195 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699771 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:56:33.716195 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699775 4951 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:56:33.716195 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699778 4951 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:56:33.716195 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699780 4951 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:56:33.716195 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699783 4951 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:56:33.716195 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699786 4951 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:56:33.716195 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699788 4951 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:56:33.716195 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699793 4951 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:56:33.716195 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699796 4951 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:56:33.716195 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699798 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:56:33.716195 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699801 4951 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:56:33.716195 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699804 4951 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:56:33.716694 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699807 4951 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:56:33.716694 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699809 4951 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:56:33.716694 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699812 4951 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:56:33.716694 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699815 4951 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:56:33.716694 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699818 4951 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:56:33.716694 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699822 4951 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:56:33.716694 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699826 4951 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:56:33.716694 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699829 4951 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:56:33.716694 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699832 4951 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:56:33.716694 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699834 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:56:33.716694 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699837 4951 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:56:33.716694 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699840 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:56:33.716694 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699842 4951 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:56:33.716694 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699845 4951 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:56:33.716694 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699848 4951 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:56:33.716694 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699850 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:56:33.716694 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699853 4951 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:56:33.716694 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699861 4951 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:56:33.716694 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699864 4951 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:56:33.717283 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699867 4951 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:56:33.717283 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699869 4951 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:56:33.717283 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699873 4951 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:56:33.717283 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699876 4951 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:56:33.717283 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699879 4951 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:56:33.717283 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699881 4951 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:56:33.717283 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699884 4951 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:56:33.717283 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699886 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:56:33.717283 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699891 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:56:33.717283 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699893 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:56:33.717283 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699896 4951 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:56:33.717283 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699899 4951 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:56:33.717283 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699901 4951 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:56:33.717283 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699904 4951 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:56:33.717283 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699907 4951 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:56:33.717283 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699909 4951 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:56:33.717283 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699912 4951 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:56:33.717283 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699914 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:56:33.717283 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699917 4951 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:56:33.718109 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699920 4951 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:56:33.718109 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699922 4951 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:56:33.718109 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699925 4951 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:56:33.718109 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699928 4951 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:56:33.718109 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699931 4951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:56:33.718109 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699933 4951 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:56:33.718109 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699936 4951 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:56:33.718109 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699938 4951 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:56:33.718109 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699941 4951 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:56:33.718109 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:33.699945 4951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:56:33.718109 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.700023 4951 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 06:56:33.718109 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.700998 4951 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 06:56:33.718109 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.701043 4951 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 06:56:33.718109 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.701209 4951 server.go:1014] "Starting client certificate rotation" Oct 30 06:56:33.718109 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.701346 4951 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 06:56:33.718109 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.701384 4951 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 06:50:53 +0000 UTC" deadline="2028-05-31 04:11:39.946719066 +0000 UTC" Oct 30 06:56:33.718837 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.701415 4951 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13893h15m6.245306085s" Oct 30 06:56:33.718837 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.701721 4951 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 06:56:33.718837 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.702830 4951 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 06:56:33.718837 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.704036 4951 log.go:25] "Validated CRI v1 runtime API" Oct 30 06:56:33.718837 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.705653 4951 log.go:25] "Validated CRI v1 image API" Oct 30 06:56:33.718837 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.706383 4951 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 06:56:33.718837 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.708246 4951 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 964a555e-b8df-45c2-a289-84189e81329e:/dev/nvme0n1p3 e464b85c-e9bd-468a-a7d7-26a80b3b8129:/dev/nvme0n1p4] Oct 30 06:56:33.719170 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.708268 4951 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b31e106cc7fee0cc85211a463e39451f22c03f14d2a3dd1bc11229c82326b2b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b31e106cc7fee0cc85211a463e39451f22c03f14d2a3dd1bc11229c82326b2b/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0eb6042ac52c60a5e6ea86f01e3f88c5940346322cb6aebc1d2c6cd7887c2634/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0eb6042ac52c60a5e6ea86f01e3f88c5940346322cb6aebc1d2c6cd7887c2634/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1489b72942e5e7d0d5e7633a8c481841dd3fc31b4a356992b62ac27bbd2566c0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1489b72942e5e7d0d5e7633a8c481841dd3fc31b4a356992b62ac27bbd2566c0/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2b91b75787a17708af821dc397864d622c6d6d570031d443fd6341c5121dab71/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2b91b75787a17708af821dc397864d622c6d6d570031d443fd6341c5121dab71/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2dc0083e51d76ca707b96c6a23ad632cb82d9d1c2cd39cc753e328e6dd78ff92/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2dc0083e51d76ca707b96c6a23ad632cb82d9d1c2cd39cc753e328e6dd78ff92/userdata/shm major:0 minor:250 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d25172907c874087f97ffa70595329ed316947772248929150c3ac4dd0845e0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d25172907c874087f97ffa70595329ed316947772248929150c3ac4dd0845e0/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/42a48dfd0e48d457bf0a01074309f6ebb34fd0abe78e8c5dc2ef2c854b221457/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/42a48dfd0e48d457bf0a01074309f6ebb34fd0abe78e8c5dc2ef2c854b221457/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5c7340e8e35fd6770010c21ea4f96f69e807d4a7ec693bc5ac261d0a2dc3fb36/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5c7340e8e35fd6770010c21ea4f96f69e807d4a7ec693bc5ac261d0a2dc3fb36/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d09172da1dab5fb0a61df183babe77f540ac253f67a3ed724be7ce751abd60b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d09172da1dab5fb0a61df183babe77f540ac253f67a3ed724be7ce751abd60b/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/749c81ba05deb3f15874314ed098daaf1fcede80c2304422e43619f8e5253292/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/749c81ba05deb3f15874314ed098daaf1fcede80c2304422e43619f8e5253292/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/75c0d71e22dcc3d3037d884ef030bdfd57b2dafb30ec4aeacc323020d0917b53/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/75c0d71e22dcc3d3037d884ef030bdfd57b2dafb30ec4aeacc323020d0917b53/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7b8eac73c37aae02e42e006a6febb299eaa9e2812de39c0b1c0e87845301168f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7b8eac73c37aae02e42e006a6febb299eaa9e2812de39c0b1c0e87845301168f/userdata/shm major:0 minor:215 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a543b8d414ca434a8d735c2bdcc4e54a60233f3dc0b601715f9647d4168ffc5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a543b8d414ca434a8d735c2bdcc4e54a60233f3dc0b601715f9647d4168ffc5/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ac1b866b9253e04ea641033e8e2583df1f52d0d1795eb4663b1407406e04a8c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ac1b866b9253e04ea641033e8e2583df1f52d0d1795eb4663b1407406e04a8c/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/95e0179d73a99531da9cb392bfc0a488a6ec96b3ae881566ae7f46acd6766e53/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/95e0179d73a99531da9cb392bfc0a488a6ec96b3ae881566ae7f46acd6766e53/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d6977b58f98e79134af3e36fac90409d296cb783b93165512a4aedc3442283f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d6977b58f98e79134af3e36fac90409d296cb783b93165512a4aedc3442283f/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b3c9bd7341cd590db1a41a04423e023234070dc4420f5a31c011e07d736ac772/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b3c9bd7341cd590db1a41a04423e023234070dc4420f5a31c011e07d736ac772/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b55fd4cb6b13ceba4c7bb3248705871e89a027d215528df2338601bab68c0470/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b55fd4cb6b13ceba4c7bb3248705871e89a027d215528df2338601bab68c0470/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b8602a9671a4e9c72f2110e3717c659f8ba28d538b2f1fb67fc540e9c952ba04/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b8602a9671a4e9c72f2110e3717c659f8ba28d538b2f1fb67fc540e9c952ba04/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ca4c81fbca9c9457378519a84eb368f9d3124ad49a53a905b44508852582cec1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ca4c81fbca9c9457378519a84eb368f9d3124ad49a53a905b44508852582cec1/userdata/shm major:0 minor:73 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d28b9134ddbbda474c10a8fe5e7d167db5e1dc7f40556a8a82e354ec532eda0d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d28b9134ddbbda474c10a8fe5e7d167db5e1dc7f40556a8a82e354ec532eda0d/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f5a34dd0a3c15a1392d3608321d2bb54b9e45b18b84a0ff3a7a644c216bdd584/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f5a34dd0a3c15a1392d3608321d2bb54b9e45b18b84a0ff3a7a644c216bdd584/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fcc1d468d5e269d20f05df58a51e5fad95b30937d4608421d3ec434933576af1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fcc1d468d5e269d20f05df58a51e5fad95b30937d4608421d3ec434933576af1/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5/volumes/kubernetes.io~projected/kube-api-access-fbzjd:{mountpoint:/var/lib/kubelet/pods/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5/volumes/kubernetes.io~projected/kube-api-access-fbzjd major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0994513f-0dac-4106-8bfe-5a89cbc11f02/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/0994513f-0dac-4106-8bfe-5a89cbc11f02/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0994513f-0dac-4106-8bfe-5a89cbc11f02/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/0994513f-0dac-4106-8bfe-5a89cbc11f02/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0994513f-0dac-4106-8bfe-5a89cbc11f02/volumes/kubernetes.io~projected/kube-api-access-x9jzs:{mountpoint:/var/lib/kubelet/pods/0994513f-0dac-4106-8bfe-5a89cbc11f02/volumes/kubernetes.io~projected/kube-api-access-x9jzs major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7/volumes/kubernetes.io~projected/kube-api-access-4vrh9:{mountpoint:/var/lib/kubelet/pods/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7/volumes/kubernetes.io~projected/kube-api-access-4vrh9 major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7/volumes/kubernetes.io~secret/default-certificate major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7/volumes/kubernetes.io~secret/stats-auth major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1327b481-6353-4d2d-8847-caa7fc7e3a3d/volumes/kubernetes.io~projected/kube-api-access-jldj6:{mountpoint:/var/lib/kubelet/pods/1327b481-6353-4d2d-8847-caa7fc7e3a3d/volumes/kubernetes.io~projected/kube-api-access-jldj6 major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14b28600-2ad3-47ff-a157-bcfacc6ab2ff/volumes/kubernetes.io~projected/kube-api-access-z6q49:{mountpoint:/var/lib/kubelet/pods/14b28600-2ad3-47ff-a157-bcfacc6ab2ff/volumes/kubernetes.io~projected/kube-api-access-z6q49 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16e1a28b-a156-4d4e-820f-8712616b4eb2/volumes/kubernetes.io~projected/kube-api-access-x7ndt:{mountpoint:/var/lib/kubelet/pods/16e1a28b-a156-4d4e-820f-8712616b4eb2/volumes/kubernetes.io~projected/kube-api-access-x7ndt major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ab9750b-0582-4c7c-b770-b1b7fbe8994c/volumes/kubernetes.io~projected/kube-api-access-jx6zb:{mountpoint:/var/lib/kubelet/pods/1ab9750b-0582-4c7c-b770-b1b7fbe8994c/volumes/kubernetes.io~projected/kube-api-access-jx6zb major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ab9750b-0582-4c7c-b770-b1b7fbe8994c/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/1ab9750b-0582-4c7c-b770-b1b7fbe8994c/volumes/kubernetes.io~secret/cert major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20a9a026-75f0-4e90-b66a-066d3b80facf/volumes/kubernetes.io~projected/kube-api-access-67hxp:{mountpoint:/var/lib/kubelet/pods/20a9a026-75f0-4e90-b66a-066d3b80facf/volumes/kubernetes.io~projected/kube-api-access-67hxp major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20a9a026-75f0-4e90-b66a-066d3b80facf/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/20a9a026-75f0-4e90-b66a-066d3b80facf/volumes/kubernetes.io~secret/serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/239195cf-db90-4e49-9064-8f7aa4af9b3a/volumes/kubernetes.io~projected/kube-api-access-4g5kc:{mountpoint:/var/lib/kubelet/pods/239195cf-db90-4e49-9064-8f7aa4af9b3a/volumes/kubernetes.io~projected/kube-api-access-4g5kc major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35296121-39f4-4fae-ae2f-f92790ad3f5c/volumes/kubernetes.io~projected/kube-api-access-945vk:{mountpoint:/var/lib/kubelet/pods/35296121-39f4-4fae-ae2f-f92790ad3f5c/volumes/kubernetes.io~projected/kube-api-access-945vk major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36b00a60-14fb-4364-8943-6bcacc5365cc/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/36b00a60-14fb-4364-8943-6bcacc5365cc/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36b00a60-14fb-4364-8943-6bcacc5365cc/volumes/kubernetes.io~projected/kube-api-access-s88rb:{mountpoint:/var/lib/kubelet/pods/36b00a60-14fb-4364-8943-6bcacc5365cc/volumes/kubernetes.io~projected/kube-api-access-s88rb major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36b00a60-14fb-4364-8943-6bcacc5365cc/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/36b00a60-14fb-4364-8943-6bcacc5365cc/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36b00a60-14fb-4364-8943-6bcacc5365cc/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/36b00a60-14fb-4364-8943-6bcacc5365cc/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce/volumes/kubernetes.io~projected/kube-api-access-xbp46:{mountpoint:/var/lib/kubelet/pods/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce/volumes/kubernetes.io~projected/kube-api-access-xbp46 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42802760-36a8-4b79-8690-5a00528503c7/volumes/kubernetes.io~projected/kube-api-access-zbpxj:{mountpoint:/var/lib/kubelet/pods/42802760-36a8-4b79-8690-5a00528503c7/volumes/kubernetes.io~projected/kube-api-access-zbpxj major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42802760-36a8-4b79-8690-5a00528503c7/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/42802760-36a8-4b79-8690-5a00528503c7/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4664d551-2cbf-42c5-a730-0d8c088f4f0f/volumes/kubernetes.io~projected/kube-api-access-t6527:{mountpoint:/var/lib/kubelet/pods/4664d551-2cbf-42c5-a730-0d8c088f4f0f/volumes/kubernetes.io~projected/kube-api-access-t6527 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b59e668-d5a4-4ce0-a1e9-a16d63f51990/volumes/kubernetes.io~projected/kube-api-access-z9n88:{mountpoint:/var/lib/kubelet/pods/4b59e668-d5a4-4ce0-a1e9-a16d63f51990/volumes/kubernetes.io~projected/kube-api-access-z9n88 major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b59e668-d5a4-4ce0-a1e9-a16d63f51990/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/4b59e668-d5a4-4ce0-a1e9-a16d63f51990/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3/volumes/kubernetes.io~projected/kube-api-access-2tf48:{mountpoint:/var/lib/kubelet/pods/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3/volumes/kubernetes.io~projected/kube-api-access-2tf48 major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50ab5bd7-ea08-460b-9932-1dac54f3005b/volumes/kubernetes.io~projected/kube-api-access-h86bq:{mountpoint:/var/lib/kubelet/pods/50ab5bd7-ea08-460b-9932-1dac54f3005b/volumes/kubernetes.io~projected/kube-api-access-h86bq major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5225f97b-21b7-453c-8d19-ccfce5ee6d54/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/5225f97b-21b7-453c-8d19-ccfce5ee6d54/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/584e4eb5-6e5e-4f96-b97a-eb58fd762347/volumes/kubernetes.io~projected/kube-api-access-984ts:{mountpoint:/var/lib/kubelet/pods/584e4eb5-6e5e-4f96-b97a-eb58fd762347/volumes/kubernetes.io~projected/kube-api-access-984ts major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f93e69e-f48e-4c0a-ab3d-655739b6a8ae/volumes/kubernetes.io~projected/kube-api-access-4bth9:{mountpoint:/var/lib/kubelet/pods/8f93e69e-f48e-4c0a-ab3d-655739b6a8ae/volumes/kubernetes.io~projected/kube-api-access-4bth9 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2/volumes/kubernetes.io~projected/kube-api-access-gp7gc:{mountpoint:/var/lib/kubelet/pods/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2/volumes/kubernetes.io~projected/kube-api-access-gp7gc major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4aa6b79-3b44-4b45-834c-78ee0fb91e02/volumes/kubernetes.io~projected/kube-api-access-2bx54:{mountpoint:/var/lib/kubelet/pods/a4aa6b79-3b44-4b45-834c-78ee0fb91e02/volumes/kubernetes.io~projected/kube-api-access-2bx54 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9d1faf4-274f-49c8-9340-f85463840b27/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/b9d1faf4-274f-49c8-9340-f85463840b27/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9d1faf4-274f-49c8-9340-f85463840b27/volumes/kubernetes.io~projected/kube-api-access-rrc6h:{mountpoint:/var/lib/kubelet/pods/b9d1faf4-274f-49c8-9340-f85463840b27/volumes/kubernetes.io~projected/kube-api-access-rrc6h major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9d1faf4-274f-49c8-9340-f85463840b27/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/b9d1faf4-274f-49c8-9340-f85463840b27/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9d1faf4-274f-49c8-9340-f85463840b27/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/b9d1faf4-274f-49c8-9340-f85463840b27/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbda0b6e-80fb-4964-a0a9-23c69a84889e/volumes/kubernetes.io~projected/kube-api-access-zj7gf:{mountpoint:/var/lib/kubelet/pods/bbda0b6e-80fb-4964-a0a9-23c69a84889e/volumes/kubernetes.io~projected/kube-api-access-zj7gf major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd315a45-3a31-44f2-9bf1-6d43df8e0c44/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/bd315a45-3a31-44f2-9bf1-6d43df8e0c44/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd315a45-3a31-44f2-9bf1-6d43df8e0c44/volumes/kubernetes.io~projected/kube-api-access-2l5w2:{mountpoint:/var/lib/kubelet/pods/bd315a45-3a31-44f2-9bf1-6d43df8e0c44/volumes/kubernetes.io~projected/kube-api-access-2l5w2 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd315a45-3a31-44f2-9bf1-6d43df8e0c44/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/bd315a45-3a31-44f2-9bf1-6d43df8e0c44/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5f1f83f-9bfd-45f5-b065-ba95cf659956/volumes/kubernetes.io~projected/kube-api-access-qm5vg:{mountpoint:/var/lib/kubelet/pods/c5f1f83f-9bfd-45f5-b065-ba95cf659956/volumes/kubernetes.io~projected/kube-api-access-qm5vg major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5f1f83f-9bfd-45f5-b065-ba95cf659956/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/c5f1f83f-9bfd-45f5-b065-ba95cf659956/volumes/kubernetes.io~secret/serving-cert major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1b84f62-7f09-42d0-8ed1-a2447efa3bf3/volumes/kubernetes.io~projected/kube-api-access-tlg4z:{mountpoint:/var/lib/kubelet/pods/d1b84f62-7f09-42d0-8ed1-a2447efa3bf3/volumes/kubernetes.io~projected/kube-api-access-tlg4z major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d930accb-e4d8-404b-8155-1b6a8edd812e/volumes/kubernetes.io~projected/kube-api-access-lgbhv:{mountpoint:/var/lib/kubelet/pods/d930accb-e4d8-404b-8155-1b6a8edd812e/volumes/kubernetes.io~projected/kube-api-access-lgbhv major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1b14112-6c97-4bfa-8f2a-5988f59ad06a/volumes/kubernetes.io~projected/kube-api-access-6v8q5:{mountpoint:/var/lib/kubelet/pods/f1b14112-6c97-4bfa-8f2a-5988f59ad06a/volumes/kubernetes.io~projected/kube-api-access-6v8q5 major:0 minor:62 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/774c7508da7cf883e0b40a00974aede9d489ffc93d6bef0962059a8faa8f5324/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/2ab11ca4761bb6f21087abad13de7039ed6421d0bcda805c6a2c592a59390296/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/714229c568d14cb9e430a6048be33a76a1b9324f2e407e1450840c400fdbd25b/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/92d4d1cc7a7cd8b28dce48456629463059443d3001a53943fc7863f7379515ca/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/c89d063aca62c9fbbc33035622eb388ac0e7113e2738c13ebccdb50ccf9ae223/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/3da175b80525747d2a0801ea040e39706f59b8628728f97b608f907338fed880/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/1228451b05633a7904e1b46367fc00253957e0dd01b7afc9be15d88c8d7ac7a1/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/1678a3e88988b1be7d2496c748172f9af784a45e7fb27cc0e47142d894eb3e7e/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/75b588ad8fb069a76d9c019724b9451604f09baa1e088f0f2a2e71f1d98dcb33/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/972c549bd05d8b52752e4b541137904744171992fea5da25e5f4c0807ea3da0a/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/b053045e148b86b92203042bb3851251277e58c7aff64312d7bf2e9f682b02ee/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/887ebf74bd376c6defd5fbe1e5199121d1fce9e0d497e2a692bb688c4ae81edf/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/93ec2f2120e5b4e6364d244cbc41de3655b341e8db53a21768699824882238d8/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/7384d649e3c708ba588785151ef6ca0c64dfc0bf3627c0b4c2eaab57216b05d0/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/e247efefa966e3a8df6649a2628e65d1463e86077dd58bdd133f860b7552960f/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/c2de3e3b94e1bcf0106234825ecefcef2dcc9dfbf8e8c0d7eeb9aa7540f64943/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/116d52881ac946289e8f8b11e95233d28b52bd128cf353ed647edbd6091a9089/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/54115fe606ba7dad0629fbe434a1bbdc43b8ee0846e0fa3f4a81f1cf7fada326/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/7490c75a3810e037b79a98268c54694e7e60b1675758a68dff673045620220c3/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/e8294f5b443688a0b0f1bc3720aad31935a0a9ceb97c872a80f08de37a575d63/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/2dd5f7b43a8aabfa3466670827d63967f8c312e0bf82bb47d38ca0305565e3a1/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/91b440a837a50c99c3b4e31bb1a6115cdbb13019fe7f51483cbc2b28b7747931/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/19a9a8bf7321b7b3fdae039904d318492985563e8044216f212986d97ea53318/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/5efb7b687be48835fccea8c762c8274d7ad01e247acf3b7d170b2e11a6d37926/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/3a611a2c7992c830b4e8762c5c0fac666de3263f2fc5b3ef47f16b4b2e38b714/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/a4fae47ca163be611b1a22d6d2ea47d5c1cac6536f4b6c7ee0d34c39bb64457e/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/35a8b3f972ece20359a66645ed78569cc1e1a6d13a082da7bc6af7d04494cad1/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/c76767bc7f4f2caab015442a4b89beed3aa9f1254fc8a7c5aa8fc87ef59a2b05/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/9ae690c08bdc01acb988a568dd484d852a40e30481dea20bd8c68368bc283f39/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/08a625800045625a2758de61af278987cd1eb61932e8e17b50690950daf2008a/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/01f5739f80e306e86db7b4551cbac62155cf0902e9e0a30774fd320447cda11b/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/d6f746251847526859b9d18195ecddbd5070f82cb1f7bf67f19d758d71bd51ab/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/50e2a58cbf2b3ecb2866737440f805c5ba367fac33e57619c99dc820c97c1dd8/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/bc8f888247eca2215bff50ccd6c275fd1f65110d3f4180b33a963f3a06cee120/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/30923a38cfea62422b611c1e28c76de65e26219cb575c1c1363460cacb53d12e/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/352071a7b924c14c11c9cdaf36ee4f446e5b26e8c8d6c9ac8cb0573fa3da0bb8/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/db219e4bcafab60b61809a5df06b1bc2640853d365ceabd674c12026e2fc3ef5/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/a4e33579a4ddbf38930d6809863d5f820ae03952a1a9503a22dc0a4f4ff6d6c3/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/75cb561921995548e9b81109c3077e5f981dc161947f8b45c1f9bbf73873cc9a/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/b5c38e2a2b6dfe97ffba2f47f660e1de23d367bba6916cfc81d472d53c911a00/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/59eadd4cc0b3fcfed05737dd895be54d38210874581f6fd0dd8103548f856e47/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/7aa66ee4bfe29435db5c873cc8de98804b880d889f3056f30c49bd03ccfa4bf4/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/f57dae0bbc8455ad62402dfc824b3905ab333e6bc68a56beb1002a68b124c83d/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/721b8cb79a317db790296d792469d2ff7d808dbdefa52501675e983813c267ca/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/24569e72332804af6ddfd9a093b528a60bcf7de259c4c8cc8be0eedaffba1368/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/8a2dca32a02e88a9bfa8960cbd985c8474019de602ed81ac91251eb2c07243d7/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/bbd6f25fd83771915264185d925a53a421213ca586d38608e21c794bf35cb3cd/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/e18846725b2b693f93e4c3c643fb12be09fcf96c083be659539289e13a15bece/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/b1163ee2320f4b6ee0868e625ee895c80d9fe4b2eb587c753e197625b5f2dcba/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-75:{mountpoint:/var/lib/containers/storage/overlay/daa19abbb50033a4071c59937f14ab19baf62c3b4877fa63404c1d92f7ccefc3/merged major:0 minor:75 fsType:overlay blockSize:0} overlay_0-77:{mountpoint:/var/lib/containers/storage/overlay/4f48335421ce2bc0fdd6ae2d46e5c27b12bf61a001668a9b878ed1e7b17aa6ed/merged major:0 minor:77 fsType:overlay blockSize:0} overlay_0-81:{mountpoint:/var/lib/containers/storage/overlay/5937751c6a421bb9f0527313ffe040b237e6580ae659e3e80afc91f97db87d60/merged major:0 minor:81 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/7c84851a0d9659e25d6e620abf03bd0b368ac99fd9f5a601832e2c500051db36/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/eec2c0b338b487f093fa3d8b11c8edecfc4167785e40633090cd620808f38047/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/a174022776b3a3818e891e440264f84813c11c976a4d76b41982febb96d5076c/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/4ce619d4b097be6d3d802e1eae8fdc478e0b73431e738d0f03521e0f8e8fdeb4/merged major:0 minor:97 fsType:overlay blockSize:0}] Oct 30 06:56:33.719507 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.716587 4951 manager.go:217] Machine: {Timestamp:2026-10-30 06:56:33.716124555 +0000 UTC m=+0.059172511 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100110 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26ddaaf9ec66d8c1a4f84ef88bc59c SystemUUID:ec26ddaa-f9ec-66d8-c1a4-f84ef88bc59c BootID:c3ed987c-4d5c-44a8-b8a7-0edb9002a3b6 Filesystems:[{Device:/var/lib/kubelet/pods/36b00a60-14fb-4364-8943-6bcacc5365cc/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:228 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1327b481-6353-4d2d-8847-caa7fc7e3a3d/volumes/kubernetes.io~projected/kube-api-access-jldj6 DeviceMajor:0 DeviceMinor:339 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b3c9bd7341cd590db1a41a04423e023234070dc4420f5a31c011e07d736ac772/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5/volumes/kubernetes.io~projected/kube-api-access-fbzjd DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/20a9a026-75f0-4e90-b66a-066d3b80facf/volumes/kubernetes.io~projected/kube-api-access-67hxp DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ac1b866b9253e04ea641033e8e2583df1f52d0d1795eb4663b1407406e04a8c/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0994513f-0dac-4106-8bfe-5a89cbc11f02/volumes/kubernetes.io~projected/kube-api-access-x9jzs DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/20a9a026-75f0-4e90-b66a-066d3b80facf/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c5f1f83f-9bfd-45f5-b065-ba95cf659956/volumes/kubernetes.io~projected/kube-api-access-qm5vg DeviceMajor:0 DeviceMinor:230 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b8602a9671a4e9c72f2110e3717c659f8ba28d538b2f1fb67fc540e9c952ba04/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5225f97b-21b7-453c-8d19-ccfce5ee6d54/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/239195cf-db90-4e49-9064-8f7aa4af9b3a/volumes/kubernetes.io~projected/kube-api-access-4g5kc DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d6977b58f98e79134af3e36fac90409d296cb783b93165512a4aedc3442283f/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2dc0083e51d76ca707b96c6a23ad632cb82d9d1c2cd39cc753e328e6dd78ff92/userdata/shm DeviceMajor:0 DeviceMinor:250 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/16e1a28b-a156-4d4e-820f-8712616b4eb2/volumes/kubernetes.io~projected/kube-api-access-x7ndt DeviceMajor:0 DeviceMinor:293 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a4aa6b79-3b44-4b45-834c-78ee0fb91e02/volumes/kubernetes.io~projected/kube-api-access-2bx54 DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fcc1d468d5e269d20f05df58a51e5fad95b30937d4608421d3ec434933576af1/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c5f1f83f-9bfd-45f5-b065-ba95cf659956/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1ab9750b-0582-4c7c-b770-b1b7fbe8994c/volumes/kubernetes.io~projected/kube-api-access-jx6zb DeviceMajor:0 DeviceMinor:239 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b9d1faf4-274f-49c8-9340-f85463840b27/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/36b00a60-14fb-4364-8943-6bcacc5365cc/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/75c0d71e22dcc3d3037d884ef030bdfd57b2dafb30ec4aeacc323020d0917b53/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4b59e668-d5a4-4ce0-a1e9-a16d63f51990/volumes/kubernetes.io~projected/kube-api-access-z9n88 DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/36b00a60-14fb-4364-8943-6bcacc5365cc/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1ab9750b-0582-4c7c-b770-b1b7fbe8994c/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:237 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/35296121-39f4-4fae-ae2f-f92790ad3f5c/volumes/kubernetes.io~projected/kube-api-access-945vk DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b9d1faf4-274f-49c8-9340-f85463840b27/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:231 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/584e4eb5-6e5e-4f96-b97a-eb58fd762347/volumes/kubernetes.io~projected/kube-api-access-984ts DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f1b14112-6c97-4bfa-8f2a-5988f59ad06a/volumes/kubernetes.io~projected/kube-api-access-6v8q5 DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-75 DeviceMajor:0 DeviceMinor:75 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d25172907c874087f97ffa70595329ed316947772248929150c3ac4dd0845e0/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/42a48dfd0e48d457bf0a01074309f6ebb34fd0abe78e8c5dc2ef2c854b221457/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bd315a45-3a31-44f2-9bf1-6d43df8e0c44/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/50ab5bd7-ea08-460b-9932-1dac54f3005b/volumes/kubernetes.io~projected/kube-api-access-h86bq DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d930accb-e4d8-404b-8155-1b6a8edd812e/volumes/kubernetes.io~projected/kube-api-access-lgbhv DeviceMajor:0 DeviceMinor:238 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2b91b75787a17708af821dc397864d622c6d6d570031d443fd6341c5121dab71/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ca4c81fbca9c9457378519a84eb368f9d3124ad49a53a905b44508852582cec1/userdata/shm DeviceMajor:0 DeviceMinor:73 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8f93e69e-f48e-4c0a-ab3d-655739b6a8ae/volumes/kubernetes.io~projected/kube-api-access-4bth9 DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7/volumes/kubernetes.io~projected/kube-api-access-4vrh9 DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/95e0179d73a99531da9cb392bfc0a488a6ec96b3ae881566ae7f46acd6766e53/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7b8eac73c37aae02e42e006a6febb299eaa9e2812de39c0b1c0e87845301168f/userdata/shm DeviceMajor:0 DeviceMinor:215 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/0994513f-0dac-4106-8bfe-5a89cbc11f02/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-81 DeviceMajor:0 DeviceMinor:81 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b9d1faf4-274f-49c8-9340-f85463840b27/volumes/kubernetes.io~projected/kube-api-access-rrc6h DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1489b72942e5e7d0d5e7633a8c481841dd3fc31b4a356992b62ac27bbd2566c0/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0994513f-0dac-4106-8bfe-5a89cbc11f02/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0eb6042ac52c60a5e6ea86f01e3f88c5940346322cb6aebc1d2c6cd7887c2634/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4b59e668-d5a4-4ce0-a1e9-a16d63f51990/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d09172da1dab5fb0a61df183babe77f540ac253f67a3ed724be7ce751abd60b/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2/volumes/kubernetes.io~projected/kube-api-access-gp7gc DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b9d1faf4-274f-49c8-9340-f85463840b27/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/36b00a60-14fb-4364-8943-6bcacc5365cc/volumes/kubernetes.io~projected/kube-api-access-s88rb DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-77 DeviceMajor:0 DeviceMinor:77 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/42802760-36a8-4b79-8690-5a00528503c7/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f5a34dd0a3c15a1392d3608321d2bb54b9e45b18b84a0ff3a7a644c216bdd584/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3/volumes/kubernetes.io~projected/kube-api-access-2tf48 DeviceMajor:0 DeviceMinor:334 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd315a45-3a31-44f2-9bf1-6d43df8e0c44/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5c7340e8e35fd6770010c21ea4f96f69e807d4a7ec693bc5ac261d0a2dc3fb36/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/749c81ba05deb3f15874314ed098daaf1fcede80c2304422e43619f8e5253292/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/42802760-36a8-4b79-8690-5a00528503c7/volumes/kubernetes.io~projected/kube-api-access-zbpxj DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b31e106cc7fee0cc85211a463e39451f22c03f14d2a3dd1bc11229c82326b2b/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d1b84f62-7f09-42d0-8ed1-a2447efa3bf3/volumes/kubernetes.io~projected/kube-api-access-tlg4z DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce/volumes/kubernetes.io~projected/kube-api-access-xbp46 DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a543b8d414ca434a8d735c2bdcc4e54a60233f3dc0b601715f9647d4168ffc5/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b55fd4cb6b13ceba4c7bb3248705871e89a027d215528df2338601bab68c0470/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bd315a45-3a31-44f2-9bf1-6d43df8e0c44/volumes/kubernetes.io~projected/kube-api-access-2l5w2 DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bbda0b6e-80fb-4964-a0a9-23c69a84889e/volumes/kubernetes.io~projected/kube-api-access-zj7gf DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/14b28600-2ad3-47ff-a157-bcfacc6ab2ff/volumes/kubernetes.io~projected/kube-api-access-z6q49 DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/4664d551-2cbf-42c5-a730-0d8c088f4f0f/volumes/kubernetes.io~projected/kube-api-access-t6527 DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d28b9134ddbbda474c10a8fe5e7d167db5e1dc7f40556a8a82e354ec532eda0d/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0b31e106cc7fee0 MacAddress:52:48:cb:18:9b:ec Speed:10000 Mtu:8901} {Name:2b91b75787a1770 MacAddress:fa:cd:c3:19:8e:61 Speed:10000 Mtu:8901} {Name:2dc0083e51d76ca MacAddress:06:f4:6e:ea:ec:9c Speed:10000 Mtu:8901} {Name:3d25172907c8740 MacAddress:4e:95:39:8b:e0:0f Speed:10000 Mtu:8901} {Name:42a48dfd0e48d45 MacAddress:ba:d2:25:5e:c1:ef Speed:10000 Mtu:8901} {Name:75c0d71e22dcc3d MacAddress:b6:85:d8:24:74:f4 Speed:10000 Mtu:8901} {Name:7b8eac73c37aae0 MacAddress:a6:83:81:2f:81:b0 Speed:10000 Mtu:8901} {Name:8ac1b866b9253e0 MacAddress:7a:bc:3a:0d:c5:c0 Speed:10000 Mtu:8901} {Name:95e0179d73a9953 MacAddress:b2:df:52:b7:b1:77 Speed:10000 Mtu:8901} {Name:9d6977b58f98e79 MacAddress:2e:77:69:29:0e:42 Speed:10000 Mtu:8901} {Name:b55fd4cb6b13ceb MacAddress:7a:c4:28:d7:5e:f6 Speed:10000 Mtu:8901} {Name:b8602a9671a4e9c MacAddress:6e:70:fb:cf:9c:71 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d6:0c:b7:c3 Speed:0 Mtu:9001} {Name:br-int MacAddress:a6:c3:9c:e7:12:b6 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:d6:0c:b7:c3 Speed:0 Mtu:9001} {Name:f5a34dd0a3c15a1 MacAddress:16:f9:2e:b5:db:83 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:36:ff:85:34:b2:37 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:fa:47:e0:7a:ce:09 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 06:56:33.719507 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.716929 4951 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 06:56:33.719507 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.716980 4951 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 06:56:33.719507 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.717327 4951 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 06:56:33.719507 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.717361 4951 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-173.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 06:56:33.719507 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.717542 4951 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 06:56:33.719507 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.717554 4951 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 06:56:33.719507 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.717592 4951 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 06:56:33.719507 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.717620 4951 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 06:56:33.719507 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.717652 4951 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 06:56:33.719507 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.717855 4951 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 06:56:33.719507 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.717999 4951 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 06:56:33.719507 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.718042 4951 kubelet.go:493] "Attempting to sync node with API server" Oct 30 06:56:33.719507 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.718060 4951 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 06:56:33.719507 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.718072 4951 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 06:56:33.719507 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.718085 4951 kubelet.go:405] "Adding apiserver pod source" Oct 30 06:56:33.719507 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.718096 4951 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 06:56:33.719507 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.719053 4951 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 06:56:33.719507 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.719066 4951 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 06:56:33.720295 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.719521 4951 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 06:56:33.721768 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.721754 4951 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 06:56:33.721768 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.721767 4951 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 06:56:33.721867 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.721858 4951 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 06:56:33.722053 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.722042 4951 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 06:56:33.722084 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.722071 4951 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 06:56:33.722195 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.722183 4951 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 06:56:33.722241 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.722204 4951 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 06:56:33.722241 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.722214 4951 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 06:56:33.722241 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.722221 4951 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 06:56:33.722241 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.722229 4951 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 06:56:33.722241 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.722235 4951 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 06:56:33.722241 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.722240 4951 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 06:56:33.722454 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.722246 4951 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 06:56:33.722454 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.722259 4951 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 06:56:33.722454 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.722265 4951 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 06:56:33.722454 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.722273 4951 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 06:56:33.722454 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.722282 4951 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 06:56:33.722454 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.722308 4951 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 06:56:33.722454 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.722315 4951 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 06:56:33.725839 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.725825 4951 server.go:1263] "Started kubelet" Oct 30 06:56:33.726250 ip-10-0-132-173 systemd[1]: Started Kubernetes Kubelet. Oct 30 06:56:33.727536 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.727097 4951 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 06:56:33.727536 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.727182 4951 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 06:56:33.727536 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.727494 4951 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 06:56:33.727728 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.727620 4951 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 06:56:33.730695 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.729735 4951 server.go:317] "Adding debug handlers to kubelet server" Oct 30 06:56:33.731274 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.731257 4951 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 06:56:33.732056 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.731553 4951 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 06:56:33.732056 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.731626 4951 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 06:50:55 +0000 UTC" deadline="2028-07-30 18:02:59.290485104 +0000 UTC" Oct 30 06:56:33.732056 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.731664 4951 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15347h6m25.558824691s" Oct 30 06:56:33.732056 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.731794 4951 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 06:56:33.732056 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.731811 4951 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 06:56:33.732056 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.731929 4951 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 06:56:33.732056 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:33.731969 4951 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 06:56:33.732410 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.732397 4951 factory.go:55] Registering systemd factory Oct 30 06:56:33.732459 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.732418 4951 factory.go:223] Registration of the systemd container factory successfully Oct 30 06:56:33.733736 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.733551 4951 factory.go:153] Registering CRI-O factory Oct 30 06:56:33.733736 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.733651 4951 factory.go:223] Registration of the crio container factory successfully Oct 30 06:56:33.733736 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.733701 4951 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 06:56:33.733736 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.733721 4951 factory.go:103] Registering Raw factory Oct 30 06:56:33.733736 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.733737 4951 manager.go:1199] Started watching for new ooms in manager Oct 30 06:56:33.734103 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.734083 4951 manager.go:319] Starting recovery of all containers Oct 30 06:56:33.741313 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741283 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9d1faf4-274f-49c8-9340-f85463840b27" volumeName="kubernetes.io/secret/b9d1faf4-274f-49c8-9340-f85463840b27-image-registry-private-configuration" seLinuxMountContext="" Oct 30 06:56:33.741422 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741314 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35296121-39f4-4fae-ae2f-f92790ad3f5c" volumeName="kubernetes.io/projected/35296121-39f4-4fae-ae2f-f92790ad3f5c-kube-api-access-945vk" seLinuxMountContext="" Oct 30 06:56:33.741422 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741328 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42802760-36a8-4b79-8690-5a00528503c7" volumeName="kubernetes.io/configmap/42802760-36a8-4b79-8690-5a00528503c7-trusted-ca" seLinuxMountContext="" Oct 30 06:56:33.741422 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741340 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4664d551-2cbf-42c5-a730-0d8c088f4f0f" volumeName="kubernetes.io/empty-dir/4664d551-2cbf-42c5-a730-0d8c088f4f0f-tmp-dir" seLinuxMountContext="" Oct 30 06:56:33.741422 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741352 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42802760-36a8-4b79-8690-5a00528503c7" volumeName="kubernetes.io/projected/42802760-36a8-4b79-8690-5a00528503c7-kube-api-access-zbpxj" seLinuxMountContext="" Oct 30 06:56:33.741422 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741365 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b59e668-d5a4-4ce0-a1e9-a16d63f51990" volumeName="kubernetes.io/empty-dir/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-tmp" seLinuxMountContext="" Oct 30 06:56:33.741422 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741378 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b59e668-d5a4-4ce0-a1e9-a16d63f51990" volumeName="kubernetes.io/secret/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-serving-cert" seLinuxMountContext="" Oct 30 06:56:33.741422 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741401 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d930accb-e4d8-404b-8155-1b6a8edd812e" volumeName="kubernetes.io/configmap/d930accb-e4d8-404b-8155-1b6a8edd812e-config-volume" seLinuxMountContext="" Oct 30 06:56:33.741422 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741415 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="044f4f8f-15bd-4f21-b6ca-2afa1f1171c5" volumeName="kubernetes.io/configmap/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 06:56:33.741825 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741426 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3be0bd9d-9b78-40d8-bfcc-d0a591c108ce" volumeName="kubernetes.io/configmap/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-multus-daemon-config" seLinuxMountContext="" Oct 30 06:56:33.741825 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741439 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42802760-36a8-4b79-8690-5a00528503c7" volumeName="kubernetes.io/secret/42802760-36a8-4b79-8690-5a00528503c7-serving-cert" seLinuxMountContext="" Oct 30 06:56:33.741825 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741452 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b59e668-d5a4-4ce0-a1e9-a16d63f51990" volumeName="kubernetes.io/projected/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-kube-api-access-z9n88" seLinuxMountContext="" Oct 30 06:56:33.741825 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741464 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3" volumeName="kubernetes.io/projected/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-kube-api-access-2tf48" seLinuxMountContext="" Oct 30 06:56:33.741825 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741476 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="584e4eb5-6e5e-4f96-b97a-eb58fd762347" volumeName="kubernetes.io/projected/584e4eb5-6e5e-4f96-b97a-eb58fd762347-kube-api-access-984ts" seLinuxMountContext="" Oct 30 06:56:33.741825 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741490 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbda0b6e-80fb-4964-a0a9-23c69a84889e" volumeName="kubernetes.io/configmap/bbda0b6e-80fb-4964-a0a9-23c69a84889e-iptables-alerter-script" seLinuxMountContext="" Oct 30 06:56:33.741825 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741507 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd315a45-3a31-44f2-9bf1-6d43df8e0c44" volumeName="kubernetes.io/configmap/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-ovnkube-config" seLinuxMountContext="" Oct 30 06:56:33.741825 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741520 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0994513f-0dac-4106-8bfe-5a89cbc11f02" volumeName="kubernetes.io/empty-dir/0994513f-0dac-4106-8bfe-5a89cbc11f02-tmp" seLinuxMountContext="" Oct 30 06:56:33.741825 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741532 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36b00a60-14fb-4364-8943-6bcacc5365cc" volumeName="kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-bound-sa-token" seLinuxMountContext="" Oct 30 06:56:33.741825 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741543 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3be0bd9d-9b78-40d8-bfcc-d0a591c108ce" volumeName="kubernetes.io/configmap/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-cni-binary-copy" seLinuxMountContext="" Oct 30 06:56:33.741825 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741557 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50ab5bd7-ea08-460b-9932-1dac54f3005b" volumeName="kubernetes.io/projected/50ab5bd7-ea08-460b-9932-1dac54f3005b-kube-api-access-h86bq" seLinuxMountContext="" Oct 30 06:56:33.741825 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741589 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2" volumeName="kubernetes.io/projected/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-kube-api-access-gp7gc" seLinuxMountContext="" Oct 30 06:56:33.741825 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741608 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4aa6b79-3b44-4b45-834c-78ee0fb91e02" volumeName="kubernetes.io/projected/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-kube-api-access-2bx54" seLinuxMountContext="" Oct 30 06:56:33.741825 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741621 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbda0b6e-80fb-4964-a0a9-23c69a84889e" volumeName="kubernetes.io/projected/bbda0b6e-80fb-4964-a0a9-23c69a84889e-kube-api-access-zj7gf" seLinuxMountContext="" Oct 30 06:56:33.741825 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741632 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd315a45-3a31-44f2-9bf1-6d43df8e0c44" volumeName="kubernetes.io/configmap/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-env-overrides" seLinuxMountContext="" Oct 30 06:56:33.741825 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741642 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42802760-36a8-4b79-8690-5a00528503c7" volumeName="kubernetes.io/configmap/42802760-36a8-4b79-8690-5a00528503c7-config" seLinuxMountContext="" Oct 30 06:56:33.741825 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741653 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4664d551-2cbf-42c5-a730-0d8c088f4f0f" volumeName="kubernetes.io/projected/4664d551-2cbf-42c5-a730-0d8c088f4f0f-kube-api-access-t6527" seLinuxMountContext="" Oct 30 06:56:33.741825 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741665 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b59e668-d5a4-4ce0-a1e9-a16d63f51990" volumeName="kubernetes.io/configmap/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-trusted-ca-bundle" seLinuxMountContext="" Oct 30 06:56:33.741825 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741693 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9d1faf4-274f-49c8-9340-f85463840b27" volumeName="kubernetes.io/empty-dir/b9d1faf4-274f-49c8-9340-f85463840b27-ca-trust-extracted" seLinuxMountContext="" Oct 30 06:56:33.741825 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741713 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5f1f83f-9bfd-45f5-b065-ba95cf659956" volumeName="kubernetes.io/secret/c5f1f83f-9bfd-45f5-b065-ba95cf659956-serving-cert" seLinuxMountContext="" Oct 30 06:56:33.741825 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741725 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1b84f62-7f09-42d0-8ed1-a2447efa3bf3" volumeName="kubernetes.io/configmap/d1b84f62-7f09-42d0-8ed1-a2447efa3bf3-serviceca" seLinuxMountContext="" Oct 30 06:56:33.741825 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741747 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36b00a60-14fb-4364-8943-6bcacc5365cc" volumeName="kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-kube-api-access-s88rb" seLinuxMountContext="" Oct 30 06:56:33.741825 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741760 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3be0bd9d-9b78-40d8-bfcc-d0a591c108ce" volumeName="kubernetes.io/projected/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-kube-api-access-xbp46" seLinuxMountContext="" Oct 30 06:56:33.741825 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741772 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5225f97b-21b7-453c-8d19-ccfce5ee6d54" volumeName="kubernetes.io/configmap/5225f97b-21b7-453c-8d19-ccfce5ee6d54-konnectivity-ca" seLinuxMountContext="" Oct 30 06:56:33.741825 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741792 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5225f97b-21b7-453c-8d19-ccfce5ee6d54" volumeName="kubernetes.io/secret/5225f97b-21b7-453c-8d19-ccfce5ee6d54-agent-certs" seLinuxMountContext="" Oct 30 06:56:33.741825 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741805 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f93e69e-f48e-4c0a-ab3d-655739b6a8ae" volumeName="kubernetes.io/projected/8f93e69e-f48e-4c0a-ab3d-655739b6a8ae-kube-api-access-4bth9" seLinuxMountContext="" Oct 30 06:56:33.741825 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741817 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9d1faf4-274f-49c8-9340-f85463840b27" volumeName="kubernetes.io/configmap/b9d1faf4-274f-49c8-9340-f85463840b27-registry-certificates" seLinuxMountContext="" Oct 30 06:56:33.741825 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741830 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5f1f83f-9bfd-45f5-b065-ba95cf659956" volumeName="kubernetes.io/projected/c5f1f83f-9bfd-45f5-b065-ba95cf659956-kube-api-access-qm5vg" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741841 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1327b481-6353-4d2d-8847-caa7fc7e3a3d" volumeName="kubernetes.io/projected/1327b481-6353-4d2d-8847-caa7fc7e3a3d-kube-api-access-jldj6" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741852 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ab9750b-0582-4c7c-b770-b1b7fbe8994c" volumeName="kubernetes.io/secret/1ab9750b-0582-4c7c-b770-b1b7fbe8994c-cert" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741864 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9d1faf4-274f-49c8-9340-f85463840b27" volumeName="kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-bound-sa-token" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741876 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd315a45-3a31-44f2-9bf1-6d43df8e0c44" volumeName="kubernetes.io/configmap/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-ovnkube-script-lib" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741887 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd315a45-3a31-44f2-9bf1-6d43df8e0c44" volumeName="kubernetes.io/projected/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-kube-api-access-2l5w2" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741898 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5f1f83f-9bfd-45f5-b065-ba95cf659956" volumeName="kubernetes.io/configmap/c5f1f83f-9bfd-45f5-b065-ba95cf659956-config" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741910 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d930accb-e4d8-404b-8155-1b6a8edd812e" volumeName="kubernetes.io/empty-dir/d930accb-e4d8-404b-8155-1b6a8edd812e-tmp-dir" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741922 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0994513f-0dac-4106-8bfe-5a89cbc11f02" volumeName="kubernetes.io/projected/0994513f-0dac-4106-8bfe-5a89cbc11f02-kube-api-access-x9jzs" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741933 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14b28600-2ad3-47ff-a157-bcfacc6ab2ff" volumeName="kubernetes.io/projected/14b28600-2ad3-47ff-a157-bcfacc6ab2ff-kube-api-access-z6q49" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741947 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36b00a60-14fb-4364-8943-6bcacc5365cc" volumeName="kubernetes.io/secret/36b00a60-14fb-4364-8943-6bcacc5365cc-installation-pull-secrets" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741960 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1b14112-6c97-4bfa-8f2a-5988f59ad06a" volumeName="kubernetes.io/projected/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-kube-api-access-6v8q5" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741972 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="044f4f8f-15bd-4f21-b6ca-2afa1f1171c5" volumeName="kubernetes.io/configmap/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741984 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20a9a026-75f0-4e90-b66a-066d3b80facf" volumeName="kubernetes.io/secret/20a9a026-75f0-4e90-b66a-066d3b80facf-serving-cert" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.741998 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c49a4a4-033f-4a62-8bc4-6b59531e8bf7" volumeName="kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-stats-auth" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.742009 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9d1faf4-274f-49c8-9340-f85463840b27" volumeName="kubernetes.io/secret/b9d1faf4-274f-49c8-9340-f85463840b27-installation-pull-secrets" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.742022 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="044f4f8f-15bd-4f21-b6ca-2afa1f1171c5" volumeName="kubernetes.io/projected/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-kube-api-access-fbzjd" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.742034 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c49a4a4-033f-4a62-8bc4-6b59531e8bf7" volumeName="kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-default-certificate" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.742046 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b59e668-d5a4-4ce0-a1e9-a16d63f51990" volumeName="kubernetes.io/configmap/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-service-ca-bundle" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.742057 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c49a4a4-033f-4a62-8bc4-6b59531e8bf7" volumeName="kubernetes.io/projected/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-kube-api-access-4vrh9" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.742076 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20a9a026-75f0-4e90-b66a-066d3b80facf" volumeName="kubernetes.io/projected/20a9a026-75f0-4e90-b66a-066d3b80facf-kube-api-access-67hxp" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.742088 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36b00a60-14fb-4364-8943-6bcacc5365cc" volumeName="kubernetes.io/secret/36b00a60-14fb-4364-8943-6bcacc5365cc-image-registry-private-configuration" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.742100 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b59e668-d5a4-4ce0-a1e9-a16d63f51990" volumeName="kubernetes.io/empty-dir/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-snapshots" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.742125 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20a9a026-75f0-4e90-b66a-066d3b80facf" volumeName="kubernetes.io/configmap/20a9a026-75f0-4e90-b66a-066d3b80facf-config" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.742137 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="239195cf-db90-4e49-9064-8f7aa4af9b3a" volumeName="kubernetes.io/projected/239195cf-db90-4e49-9064-8f7aa4af9b3a-kube-api-access-4g5kc" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.742149 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36b00a60-14fb-4364-8943-6bcacc5365cc" volumeName="kubernetes.io/configmap/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-certificates" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.742161 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5943f8dc-3e86-49fa-ac40-70a1f11fb69e" volumeName="kubernetes.io/configmap/5943f8dc-3e86-49fa-ac40-70a1f11fb69e-nginx-conf" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.742173 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2" volumeName="kubernetes.io/configmap/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-telemetry-config" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.742190 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ab9750b-0582-4c7c-b770-b1b7fbe8994c" volumeName="kubernetes.io/projected/1ab9750b-0582-4c7c-b770-b1b7fbe8994c-kube-api-access-jx6zb" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.742202 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36b00a60-14fb-4364-8943-6bcacc5365cc" volumeName="kubernetes.io/configmap/36b00a60-14fb-4364-8943-6bcacc5365cc-trusted-ca" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.742217 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50ab5bd7-ea08-460b-9932-1dac54f3005b" volumeName="kubernetes.io/configmap/50ab5bd7-ea08-460b-9932-1dac54f3005b-samples-operator-config" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.742229 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96024683-dd4e-4d78-b5ad-6aa9ccc4bc05" volumeName="kubernetes.io/secret/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-original-pull-secret" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.742241 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9d1faf4-274f-49c8-9340-f85463840b27" volumeName="kubernetes.io/configmap/b9d1faf4-274f-49c8-9340-f85463840b27-trusted-ca" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.742252 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd315a45-3a31-44f2-9bf1-6d43df8e0c44" volumeName="kubernetes.io/secret/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.742266 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d930accb-e4d8-404b-8155-1b6a8edd812e" volumeName="kubernetes.io/projected/d930accb-e4d8-404b-8155-1b6a8edd812e-kube-api-access-lgbhv" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.742278 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0994513f-0dac-4106-8bfe-5a89cbc11f02" volumeName="kubernetes.io/empty-dir/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-tuned" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.742289 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9d1faf4-274f-49c8-9340-f85463840b27" volumeName="kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-kube-api-access-rrc6h" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.742302 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1b84f62-7f09-42d0-8ed1-a2447efa3bf3" volumeName="kubernetes.io/projected/d1b84f62-7f09-42d0-8ed1-a2447efa3bf3-kube-api-access-tlg4z" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.742314 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="044f4f8f-15bd-4f21-b6ca-2afa1f1171c5" volumeName="kubernetes.io/configmap/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-cni-binary-copy" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.742326 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16e1a28b-a156-4d4e-820f-8712616b4eb2" volumeName="kubernetes.io/projected/16e1a28b-a156-4d4e-820f-8712616b4eb2-kube-api-access-x7ndt" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.742338 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36b00a60-14fb-4364-8943-6bcacc5365cc" volumeName="kubernetes.io/empty-dir/36b00a60-14fb-4364-8943-6bcacc5365cc-ca-trust-extracted" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.742352 4951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3" volumeName="kubernetes.io/empty-dir/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-data-volume" seLinuxMountContext="" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.742369 4951 reconstruct.go:97] "Volume reconstruction finished" Oct 30 06:56:33.743011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.742378 4951 reconciler.go:29] "Reconciler: start to sync state" Oct 30 06:56:33.749667 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.749648 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/1.log" Oct 30 06:56:33.750141 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.750122 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/1.log" Oct 30 06:56:33.759032 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.759007 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 06:56:33.759561 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.759544 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 06:56:33.761756 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.761557 4951 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 06:56:33.763413 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.763123 4951 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 06:56:33.763413 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.763154 4951 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 06:56:33.763413 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.763173 4951 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 06:56:33.763413 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:33.763205 4951 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 06:56:33.775156 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.775140 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 06:56:33.775482 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.775453 4951 generic.go:356] "Generic (PLEG): container finished" podID="bd315a45-3a31-44f2-9bf1-6d43df8e0c44" containerID="52d10454fd32559fd11d86f286c2497f3c9d8bf4db566724c2df2a9174fd166b" exitCode=1 Oct 30 06:56:33.781213 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.781190 4951 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 06:56:33.781295 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.781256 4951 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 06:56:33.799055 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.799021 4951 generic.go:356] "Generic (PLEG): container finished" podID="044f4f8f-15bd-4f21-b6ca-2afa1f1171c5" containerID="72ca7c79831f83085c6d6248110e1223874dcc00d9aac07ac2ba17a3082839e7" exitCode=0 Oct 30 06:56:33.799055 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.799048 4951 generic.go:356] "Generic (PLEG): container finished" podID="044f4f8f-15bd-4f21-b6ca-2afa1f1171c5" containerID="8a9e3a57a54719c70ece10510e62315ce8b502c7af510bb0fd91df90ca25d6e0" exitCode=0 Oct 30 06:56:33.799055 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.799058 4951 generic.go:356] "Generic (PLEG): container finished" podID="044f4f8f-15bd-4f21-b6ca-2afa1f1171c5" containerID="e4886fb1f9c4052deeb996ecb4baca4a76b47b10853941a7c51db8ce8fbc7be3" exitCode=0 Oct 30 06:56:33.799304 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.799065 4951 generic.go:356] "Generic (PLEG): container finished" podID="044f4f8f-15bd-4f21-b6ca-2afa1f1171c5" containerID="554723614f7fea2c063968e51c10eb1bff40141e57654959c121ebcb253a849f" exitCode=0 Oct 30 06:56:33.799304 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.799073 4951 generic.go:356] "Generic (PLEG): container finished" podID="044f4f8f-15bd-4f21-b6ca-2afa1f1171c5" containerID="ef9501af39d44bc8804d23d6b5cea138908726a5ec12b2fc25a648e1cb1b748d" exitCode=0 Oct 30 06:56:33.799304 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.799081 4951 generic.go:356] "Generic (PLEG): container finished" podID="044f4f8f-15bd-4f21-b6ca-2afa1f1171c5" containerID="b9d8a19cd0437c48fd36c921903a8e0b5fc23aef9b9568d9c091831a36f35f17" exitCode=0 Oct 30 06:56:33.800328 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.800301 4951 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 06:56:33.800512 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.800493 4951 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 06:56:33.803481 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.803458 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/1.log" Oct 30 06:56:33.803628 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.803490 4951 generic.go:356] "Generic (PLEG): container finished" podID="42802760-36a8-4b79-8690-5a00528503c7" containerID="ab6a3aae9beab6299cb2ae4ffc208236cde7fa979fe642aba021008f4c9e948f" exitCode=255 Oct 30 06:56:33.811736 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.811702 4951 generic.go:356] "Generic (PLEG): container finished" podID="6ad70715366d9db6ff14ef77cdc50313" containerID="851007e35aaa6fe3639c9c48fb3962da44b84fce2bbcfcbfda033d3759d2e159" exitCode=0 Oct 30 06:56:33.826741 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.826669 4951 manager.go:324] Recovery completed Oct 30 06:56:33.832041 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:33.832010 4951 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 06:56:33.844816 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.844786 4951 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 06:56:33.848482 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.848462 4951 cpu_manager.go:226] "Starting" policy="none" Oct 30 06:56:33.848482 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.848481 4951 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 06:56:33.848637 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.848506 4951 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 06:56:33.848716 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.848673 4951 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 06:56:33.848716 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.848689 4951 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 06:56:33.848716 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.848707 4951 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 06:56:33.848716 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.848713 4951 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 06:56:33.848716 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.848719 4951 policy_none.go:50] "Start" Oct 30 06:56:33.848861 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.848728 4951 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 06:56:33.848861 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.848737 4951 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 06:56:33.848911 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.848863 4951 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 06:56:33.848911 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.848871 4951 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 06:56:33.848911 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.848877 4951 policy_none.go:44] "Start" Oct 30 06:56:33.853318 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.853282 4951 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 06:56:33.855954 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.855938 4951 manager.go:341] "Starting Device Plugin manager" Oct 30 06:56:33.856555 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:33.855986 4951 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 06:56:33.856629 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.856558 4951 server.go:87] "Starting device plugin registration server" Oct 30 06:56:33.856842 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.856830 4951 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 06:56:33.856887 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.856844 4951 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 06:56:33.856995 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.856968 4951 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 06:56:33.857142 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.857103 4951 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 06:56:33.857214 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.857153 4951 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 06:56:33.857214 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.857161 4951 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 06:56:33.857295 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.857216 4951 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T06:56:33.857140066Z","UUID":"703cc5be-7661-417e-bed4-bb512b587f7c","Handler":null,"Name":"","Endpoint":""} Oct 30 06:56:33.858500 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.858479 4951 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 06:56:33.858615 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.858518 4951 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 06:56:33.858752 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:33.858728 4951 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 06:56:33.863930 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.863873 4951 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal"] Oct 30 06:56:33.865146 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.865097 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" event={"ID":"5b1dc20b8a7bd35e96a3f207cff8defc","Type":"ContainerStarted","Data":"32ce3a26e85848f3e37211dff87c5e578eefe1cb83b3a9c9875180d85156fed0"} Oct 30 06:56:33.865231 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.865158 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" event={"ID":"5b1dc20b8a7bd35e96a3f207cff8defc","Type":"ContainerStarted","Data":"b3c9bd7341cd590db1a41a04423e023234070dc4420f5a31c011e07d736ac772"} Oct 30 06:56:33.865398 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.865368 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" event={"ID":"6ad70715366d9db6ff14ef77cdc50313","Type":"ContainerStarted","Data":"d37ee28df8c2343082e62d483b7145a8aad5f31d2e4542c364beaf8940b2d106"} Oct 30 06:56:33.865481 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.865433 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" event={"ID":"6ad70715366d9db6ff14ef77cdc50313","Type":"ContainerDied","Data":"851007e35aaa6fe3639c9c48fb3962da44b84fce2bbcfcbfda033d3759d2e159"} Oct 30 06:56:33.865481 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.865450 4951 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 06:56:33.865481 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.865464 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" event={"ID":"6ad70715366d9db6ff14ef77cdc50313","Type":"ContainerStarted","Data":"749c81ba05deb3f15874314ed098daaf1fcede80c2304422e43619f8e5253292"} Oct 30 06:56:33.865643 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.865629 4951 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" Oct 30 06:56:33.893023 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.892998 4951 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 06:56:33.893161 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:33.893067 4951 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-173.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" Oct 30 06:56:33.893541 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.893527 4951 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 06:56:33.893657 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:33.893565 4951 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 06:56:33.932508 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.932481 4951 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 06:56:33.954826 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.954805 4951 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 06:56:33.954917 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:33.954847 4951 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 06:56:33.954917 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.954858 4951 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" Oct 30 06:56:33.957921 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.957906 4951 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 06:56:33.959412 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.959387 4951 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientMemory" Oct 30 06:56:33.959505 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.959422 4951 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 06:56:33.959505 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.959435 4951 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientPID" Oct 30 06:56:33.959505 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.959489 4951 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-173.ec2.internal" Oct 30 06:56:33.979565 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.979545 4951 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 06:56:33.979648 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:33.979600 4951 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-173.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" Oct 30 06:56:33.981414 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.981400 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6ad70715366d9db6ff14ef77cdc50313-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal\" (UID: \"6ad70715366d9db6ff14ef77cdc50313\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 06:56:33.981452 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.981422 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ad70715366d9db6ff14ef77cdc50313-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal\" (UID: \"6ad70715366d9db6ff14ef77cdc50313\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 06:56:33.981452 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.981438 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b1dc20b8a7bd35e96a3f207cff8defc-config\") pod \"kube-apiserver-proxy-ip-10-0-132-173.ec2.internal\" (UID: \"5b1dc20b8a7bd35e96a3f207cff8defc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" Oct 30 06:56:33.995215 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.995191 4951 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-132-173.ec2.internal" Oct 30 06:56:33.995262 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:33.995257 4951 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-173.ec2.internal" Oct 30 06:56:34.081946 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.081864 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6ad70715366d9db6ff14ef77cdc50313-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal\" (UID: \"6ad70715366d9db6ff14ef77cdc50313\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 06:56:34.081946 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.081819 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6ad70715366d9db6ff14ef77cdc50313-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal\" (UID: \"6ad70715366d9db6ff14ef77cdc50313\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 06:56:34.082116 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.081953 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ad70715366d9db6ff14ef77cdc50313-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal\" (UID: \"6ad70715366d9db6ff14ef77cdc50313\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 06:56:34.082116 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.081974 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b1dc20b8a7bd35e96a3f207cff8defc-config\") pod \"kube-apiserver-proxy-ip-10-0-132-173.ec2.internal\" (UID: \"5b1dc20b8a7bd35e96a3f207cff8defc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" Oct 30 06:56:34.082116 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.082012 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b1dc20b8a7bd35e96a3f207cff8defc-config\") pod \"kube-apiserver-proxy-ip-10-0-132-173.ec2.internal\" (UID: \"5b1dc20b8a7bd35e96a3f207cff8defc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" Oct 30 06:56:34.082116 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.082028 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ad70715366d9db6ff14ef77cdc50313-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal\" (UID: \"6ad70715366d9db6ff14ef77cdc50313\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 06:56:34.718823 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.718791 4951 apiserver.go:52] "Watching apiserver" Oct 30 06:56:34.785234 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.785208 4951 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 06:56:34.785909 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.785884 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5dcb9bff4d-5xf9f","openshift-kube-storage-version-migrator/migrator-66df5d496b-6t2h6","openshift-network-diagnostics/network-check-target-rjdmc","kube-system/kas-connection-checker-685895c7c9-zwfbr","openshift-multus/multus-q9jl6","openshift-image-registry/node-ca-9fpvq","kube-system/global-pull-secret-syncer-ts77g","openshift-multus/multus-additional-cni-plugins-t4rnr","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cflrf","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z5nm2","openshift-image-registry/image-registry-5b5f74d469-87gxs","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8xtng","openshift-dns/node-resolver-qq6f5","openshift-cluster-node-tuning-operator/tuned-qqnmt","openshift-multus/network-metrics-daemon-g6vbv","kube-system/kas-connection-checker-685895c7c9-7dwjw","openshift-monitoring/cluster-monitoring-operator-884576c57-l6qcv","kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal","openshift-network-operator/iptables-alerter-zzvr7","kube-system/konnectivity-agent-mjwrd","openshift-network-diagnostics/network-check-source-6f78d59f99-fv48w","openshift-console-operator/console-operator-7cd56b8dbc-zw4km","openshift-ovn-kubernetes/ovnkube-node-8tlbb","openshift-insights/insights-runtime-extractor-r5bpp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4","openshift-dns/dns-default-vgrqz","openshift-insights/insights-operator-7c4986d449-qqt8g","openshift-network-console/networking-console-plugin-86c478bb88-cjn9t","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal","openshift-service-ca-operator/service-ca-operator-544d749c8c-2kt77","openshift-ingress-canary/ingress-canary-qrksf","openshift-ingress/router-default-7f6bb8cb6f-fzpwq","kube-system/kas-connection-checker-685895c7c9-5bl7s"] Oct 30 06:56:34.787196 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.787180 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l6qcv" Oct 30 06:56:34.790340 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.790312 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:34.791361 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.791340 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cflrf" Oct 30 06:56:34.791991 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.791974 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:34.793508 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.793491 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-cjn9t" Oct 30 06:56:34.796105 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.796090 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:56:34.799515 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.799496 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:34.800107 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.800094 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vgrqz" Oct 30 06:56:34.801418 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.801402 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 06:56:34.801539 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.801511 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:34.801617 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.801585 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 06:56:34.802537 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.802520 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 06:56:34.802629 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.802565 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 06:56:34.802704 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.802689 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 06:56:34.802772 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.802689 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 06:56:34.802818 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.802787 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-x2j6m\"" Oct 30 06:56:34.807866 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.807851 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-mkfxw\"" Oct 30 06:56:34.818329 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.818302 4951 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 06:56:34.834740 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.834713 4951 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 06:56:34.840853 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.840837 4951 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 06:56:34.840922 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:34.840891 4951 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 06:56:34.851975 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.851948 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 06:56:34.852986 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.852962 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 06:56:34.853221 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853205 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 06:56:34.853221 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853176 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 06:56:34.853311 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853254 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 06:56:34.853348 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853309 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 06:56:34.853348 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853328 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 06:56:34.853482 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853400 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 06:56:34.853536 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853494 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-kkxlz\"" Oct 30 06:56:34.853536 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853508 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-s89sh\"" Oct 30 06:56:34.853536 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853522 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4sbng\"" Oct 30 06:56:34.853536 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853533 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 06:56:34.853733 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853552 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 06:56:34.853733 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853563 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7vgdh\"" Oct 30 06:56:34.853733 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853630 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7f8th\"" Oct 30 06:56:34.853733 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853645 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fp7lr\"" Oct 30 06:56:34.853733 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853693 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 06:56:34.853733 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853699 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 06:56:34.853733 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853693 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-25rrt\"" Oct 30 06:56:34.853733 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853720 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 06:56:34.853733 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853733 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 06:56:34.854072 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853747 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 06:56:34.854072 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853754 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r8khf\"" Oct 30 06:56:34.854072 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853773 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-7qqzj\"" Oct 30 06:56:34.854072 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853782 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 06:56:34.854072 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853816 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 06:56:34.854072 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853824 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gp96c\"" Oct 30 06:56:34.854072 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853818 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7cszt\"" Oct 30 06:56:34.854072 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853862 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-b2dtw\"" Oct 30 06:56:34.854072 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853871 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-7pxn9\"" Oct 30 06:56:34.854072 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853884 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jgr8q\"" Oct 30 06:56:34.854072 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853912 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hwx8p\"" Oct 30 06:56:34.854072 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853926 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cmb4r\"" Oct 30 06:56:34.854072 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853943 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fvc75\"" Oct 30 06:56:34.854072 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853962 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-4flrq\"" Oct 30 06:56:34.854072 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853961 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-f6r5d\"" Oct 30 06:56:34.854072 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.853985 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7mg7c\"" Oct 30 06:56:34.854072 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.854013 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zl222\"" Oct 30 06:56:34.854492 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.854098 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r9nbg\"" Oct 30 06:56:34.854492 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.854140 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-vcrml\"" Oct 30 06:56:34.854492 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.854185 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-lp7h7\"" Oct 30 06:56:34.854492 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.854266 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 06:56:34.854492 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.854291 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 06:56:34.854492 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.854319 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 06:56:34.854492 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.854352 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4hfgh\"" Oct 30 06:56:34.864043 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.864026 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 06:56:34.864344 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.864305 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 06:56:34.864344 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.864313 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 06:56:34.864344 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.864335 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 06:56:34.864508 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.864347 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 06:56:34.864508 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.864340 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 06:56:34.864508 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.864392 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 06:56:34.864508 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.864395 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 06:56:34.864508 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.864405 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 06:56:34.864508 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.864449 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 06:56:34.864508 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.864477 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 06:56:34.864818 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.864599 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 06:56:34.864818 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.864634 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 06:56:34.864818 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.864689 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 06:56:34.864818 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.864697 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 06:56:34.864818 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.864715 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 06:56:34.864818 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.864719 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 06:56:34.864818 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.864741 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 06:56:34.864818 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.864772 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 06:56:34.865144 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.864859 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 06:56:34.865144 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.864873 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 06:56:34.865144 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.864875 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 06:56:34.865144 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.864972 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 06:56:34.865144 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.864994 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 06:56:34.865144 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.865013 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 06:56:34.865144 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.865022 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 06:56:34.865144 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.865015 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 06:56:34.865144 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.865056 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 06:56:34.865144 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.865149 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 06:56:34.865510 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.865160 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 06:56:34.865510 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.865194 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 06:56:34.865510 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.865204 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 06:56:34.865510 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.865207 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 06:56:34.865510 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.865204 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 06:56:34.865510 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.865238 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 06:56:34.865510 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.865250 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 06:56:34.865510 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.865254 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 06:56:34.865510 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.865274 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 06:56:34.865510 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.865293 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 06:56:34.865510 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.865303 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 06:56:34.865510 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.865308 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 06:56:34.865510 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.865314 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 06:56:34.865510 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.865423 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 06:56:34.865510 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.865457 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 06:56:34.865510 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.865501 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 06:56:34.865994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.865552 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 06:56:34.865994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.865597 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 06:56:34.865994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.865630 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 06:56:34.865994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.865661 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 06:56:34.865994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.865660 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 06:56:34.865994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.865671 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 06:56:34.865994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.865694 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 06:56:34.866609 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.866567 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 06:56:34.878338 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.878319 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 06:56:34.878425 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.878368 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 06:56:34.878425 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.878368 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 06:56:34.885141 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.885119 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-multus-conf-dir\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.885214 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.885143 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-var-lib-openvswitch\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.885214 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.885158 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-run-netns\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.885214 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.885176 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-slash\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.885214 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.885193 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-etc-kubernetes\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.885343 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.885235 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42802760-36a8-4b79-8690-5a00528503c7-trusted-ca\") pod \"console-operator-7cd56b8dbc-zw4km\" (UID: \"42802760-36a8-4b79-8690-5a00528503c7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" Oct 30 06:56:34.885343 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.885263 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:34.885396 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.885360 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d1b84f62-7f09-42d0-8ed1-a2447efa3bf3-serviceca\") pod \"node-ca-9fpvq\" (UID: \"d1b84f62-7f09-42d0-8ed1-a2447efa3bf3\") " pod="openshift-image-registry/node-ca-9fpvq" Oct 30 06:56:34.885396 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.885387 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-service-ca-bundle\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:34.885465 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.885403 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d930accb-e4d8-404b-8155-1b6a8edd812e-config-volume\") pod \"dns-default-vgrqz\" (UID: \"d930accb-e4d8-404b-8155-1b6a8edd812e\") " pod="openshift-dns/dns-default-vgrqz" Oct 30 06:56:34.885517 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.885469 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-data-volume\") pod \"insights-runtime-extractor-r5bpp\" (UID: \"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3\") " pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:34.885517 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.885479 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42802760-36a8-4b79-8690-5a00528503c7-trusted-ca\") pod \"console-operator-7cd56b8dbc-zw4km\" (UID: \"42802760-36a8-4b79-8690-5a00528503c7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" Oct 30 06:56:34.885627 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.885539 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/50ab5bd7-ea08-460b-9932-1dac54f3005b-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-cflrf\" (UID: \"50ab5bd7-ea08-460b-9932-1dac54f3005b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cflrf" Oct 30 06:56:34.885627 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.885567 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4664d551-2cbf-42c5-a730-0d8c088f4f0f-tmp-dir\") pod \"node-resolver-qq6f5\" (UID: \"4664d551-2cbf-42c5-a730-0d8c088f4f0f\") " pod="openshift-dns/node-resolver-qq6f5" Oct 30 06:56:34.885627 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.885596 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-data-volume\") pod \"insights-runtime-extractor-r5bpp\" (UID: \"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3\") " pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:34.885627 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.885612 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-systemd-units\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.885806 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.885561 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d1b84f62-7f09-42d0-8ed1-a2447efa3bf3-serviceca\") pod \"node-ca-9fpvq\" (UID: \"d1b84f62-7f09-42d0-8ed1-a2447efa3bf3\") " pod="openshift-image-registry/node-ca-9fpvq" Oct 30 06:56:34.885806 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.885643 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-system-cni-dir\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:56:34.885806 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.885626 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d930accb-e4d8-404b-8155-1b6a8edd812e-config-volume\") pod \"dns-default-vgrqz\" (UID: \"d930accb-e4d8-404b-8155-1b6a8edd812e\") " pod="openshift-dns/dns-default-vgrqz" Oct 30 06:56:34.885806 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.885682 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4664d551-2cbf-42c5-a730-0d8c088f4f0f-tmp-dir\") pod \"node-resolver-qq6f5\" (UID: \"4664d551-2cbf-42c5-a730-0d8c088f4f0f\") " pod="openshift-dns/node-resolver-qq6f5" Oct 30 06:56:34.885806 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.885687 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-certificates\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:34.885806 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.885730 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/50ab5bd7-ea08-460b-9932-1dac54f3005b-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-cflrf\" (UID: \"50ab5bd7-ea08-460b-9932-1dac54f3005b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cflrf" Oct 30 06:56:34.885806 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.885733 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-lib-modules\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.885806 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.885780 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l6qcv\" (UID: \"8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l6qcv" Oct 30 06:56:34.886211 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.885835 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-qqt8g\" (UID: \"4b59e668-d5a4-4ce0-a1e9-a16d63f51990\") " pod="openshift-insights/insights-operator-7c4986d449-qqt8g" Oct 30 06:56:34.886211 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.885862 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-ovn-node-metrics-cert\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.886211 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.885921 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-certificates\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:34.886211 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.885974 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-host\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.886211 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.886010 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0994513f-0dac-4106-8bfe-5a89cbc11f02-tmp\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.886211 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.886048 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vrh9\" (UniqueName: \"kubernetes.io/projected/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-kube-api-access-4vrh9\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:34.886211 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.886113 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bbda0b6e-80fb-4964-a0a9-23c69a84889e-host-slash\") pod \"iptables-alerter-zzvr7\" (UID: \"bbda0b6e-80fb-4964-a0a9-23c69a84889e\") " pod="openshift-network-operator/iptables-alerter-zzvr7" Oct 30 06:56:34.886211 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.886113 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0994513f-0dac-4106-8bfe-5a89cbc11f02-tmp\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.886211 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.886135 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-multus-daemon-config\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.886211 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.886158 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-bound-sa-token\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:34.886211 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.886174 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9d1faf4-274f-49c8-9340-f85463840b27-trusted-ca\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:34.886211 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.886189 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrc6h\" (UniqueName: \"kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-kube-api-access-rrc6h\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:34.886211 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.886204 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-sysctl-conf\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.886751 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.886227 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9jzs\" (UniqueName: \"kubernetes.io/projected/0994513f-0dac-4106-8bfe-5a89cbc11f02-kube-api-access-x9jzs\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.886751 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.886252 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-metrics-certs\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:34.886751 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.886266 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-ovn-node-metrics-cert\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.886751 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.886278 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6q49\" (UniqueName: \"kubernetes.io/projected/14b28600-2ad3-47ff-a157-bcfacc6ab2ff-kube-api-access-z6q49\") pod \"kas-connection-checker-685895c7c9-7dwjw\" (UID: \"14b28600-2ad3-47ff-a157-bcfacc6ab2ff\") " pod="kube-system/kas-connection-checker-685895c7c9-7dwjw" Oct 30 06:56:34.886751 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.886305 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r5bpp\" (UID: \"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3\") " pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:34.886751 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.886330 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/50ab5bd7-ea08-460b-9932-1dac54f3005b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-cflrf\" (UID: \"50ab5bd7-ea08-460b-9932-1dac54f3005b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cflrf" Oct 30 06:56:34.886751 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.886356 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6527\" (UniqueName: \"kubernetes.io/projected/4664d551-2cbf-42c5-a730-0d8c088f4f0f-kube-api-access-t6527\") pod \"node-resolver-qq6f5\" (UID: \"4664d551-2cbf-42c5-a730-0d8c088f4f0f\") " pod="openshift-dns/node-resolver-qq6f5" Oct 30 06:56:34.886751 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.886381 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlg4z\" (UniqueName: \"kubernetes.io/projected/d1b84f62-7f09-42d0-8ed1-a2447efa3bf3-kube-api-access-tlg4z\") pod \"node-ca-9fpvq\" (UID: \"d1b84f62-7f09-42d0-8ed1-a2447efa3bf3\") " pod="openshift-image-registry/node-ca-9fpvq" Oct 30 06:56:34.886751 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.886406 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-snapshots\") pod \"insights-operator-7c4986d449-qqt8g\" (UID: \"4b59e668-d5a4-4ce0-a1e9-a16d63f51990\") " pod="openshift-insights/insights-operator-7c4986d449-qqt8g" Oct 30 06:56:34.886751 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.886414 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-multus-daemon-config\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.886751 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.886433 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9n88\" (UniqueName: \"kubernetes.io/projected/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-kube-api-access-z9n88\") pod \"insights-operator-7c4986d449-qqt8g\" (UID: \"4b59e668-d5a4-4ce0-a1e9-a16d63f51990\") " pod="openshift-insights/insights-operator-7c4986d449-qqt8g" Oct 30 06:56:34.886751 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.886491 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/20a9a026-75f0-4e90-b66a-066d3b80facf-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-8xtng\" (UID: \"20a9a026-75f0-4e90-b66a-066d3b80facf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8xtng" Oct 30 06:56:34.886751 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.886560 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-snapshots\") pod \"insights-operator-7c4986d449-qqt8g\" (UID: \"4b59e668-d5a4-4ce0-a1e9-a16d63f51990\") " pod="openshift-insights/insights-operator-7c4986d449-qqt8g" Oct 30 06:56:34.886751 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.886567 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-qqt8g\" (UID: \"4b59e668-d5a4-4ce0-a1e9-a16d63f51990\") " pod="openshift-insights/insights-operator-7c4986d449-qqt8g" Oct 30 06:56:34.886751 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.886593 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-run-systemd\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.887381 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.886973 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9d1faf4-274f-49c8-9340-f85463840b27-trusted-ca\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:34.887381 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.886651 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-log-socket\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.887381 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.887118 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-ovnkube-config\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.887381 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.887151 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5225f97b-21b7-453c-8d19-ccfce5ee6d54-konnectivity-ca\") pod \"konnectivity-agent-mjwrd\" (UID: \"5225f97b-21b7-453c-8d19-ccfce5ee6d54\") " pod="kube-system/konnectivity-agent-mjwrd" Oct 30 06:56:34.887381 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.887153 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/20a9a026-75f0-4e90-b66a-066d3b80facf-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-8xtng\" (UID: \"20a9a026-75f0-4e90-b66a-066d3b80facf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8xtng" Oct 30 06:56:34.887381 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.887184 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-system-cni-dir\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.887696 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.887465 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-ovnkube-config\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.887696 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.887482 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5225f97b-21b7-453c-8d19-ccfce5ee6d54-konnectivity-ca\") pod \"konnectivity-agent-mjwrd\" (UID: \"5225f97b-21b7-453c-8d19-ccfce5ee6d54\") " pod="kube-system/konnectivity-agent-mjwrd" Oct 30 06:56:34.887696 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.887592 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c5f1f83f-9bfd-45f5-b065-ba95cf659956-serving-cert\") pod \"service-ca-operator-544d749c8c-2kt77\" (UID: \"c5f1f83f-9bfd-45f5-b065-ba95cf659956\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2kt77" Oct 30 06:56:34.887696 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.887633 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h86bq\" (UniqueName: \"kubernetes.io/projected/50ab5bd7-ea08-460b-9932-1dac54f3005b-kube-api-access-h86bq\") pod \"cluster-samples-operator-6ddb6c8b94-cflrf\" (UID: \"50ab5bd7-ea08-460b-9932-1dac54f3005b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cflrf" Oct 30 06:56:34.887696 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.887665 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7ndt\" (UniqueName: \"kubernetes.io/projected/16e1a28b-a156-4d4e-820f-8712616b4eb2-kube-api-access-x7ndt\") pod \"network-check-target-rjdmc\" (UID: \"16e1a28b-a156-4d4e-820f-8712616b4eb2\") " pod="openshift-network-diagnostics/network-check-target-rjdmc" Oct 30 06:56:34.887696 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.887698 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-var-lib-kubelet\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.888351 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.887752 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-service-ca-bundle\") pod \"insights-operator-7c4986d449-qqt8g\" (UID: \"4b59e668-d5a4-4ce0-a1e9-a16d63f51990\") " pod="openshift-insights/insights-operator-7c4986d449-qqt8g" Oct 30 06:56:34.888351 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.887778 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-kubelet\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.888351 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.887811 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5225f97b-21b7-453c-8d19-ccfce5ee6d54-agent-certs\") pod \"konnectivity-agent-mjwrd\" (UID: \"5225f97b-21b7-453c-8d19-ccfce5ee6d54\") " pod="kube-system/konnectivity-agent-mjwrd" Oct 30 06:56:34.888351 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.887881 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-device-dir\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:56:34.888351 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.887915 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:56:34.888351 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.887949 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbzjd\" (UniqueName: \"kubernetes.io/projected/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-kube-api-access-fbzjd\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:56:34.888351 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.887996 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-kubelet-config\") pod \"global-pull-secret-syncer-ts77g\" (UID: \"96024683-dd4e-4d78-b5ad-6aa9ccc4bc05\") " pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:56:34.888351 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.888023 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bx54\" (UniqueName: \"kubernetes.io/projected/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-kube-api-access-2bx54\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:56:34.888351 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.888079 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bth9\" (UniqueName: \"kubernetes.io/projected/8f93e69e-f48e-4c0a-ab3d-655739b6a8ae-kube-api-access-4bth9\") pod \"kas-connection-checker-685895c7c9-5bl7s\" (UID: \"8f93e69e-f48e-4c0a-ab3d-655739b6a8ae\") " pod="kube-system/kas-connection-checker-685895c7c9-5bl7s" Oct 30 06:56:34.888351 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.888127 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-cni-binary-copy\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.888351 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.888156 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-hostroot\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.888351 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.888185 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:56:34.888351 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.888216 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g5kc\" (UniqueName: \"kubernetes.io/projected/239195cf-db90-4e49-9064-8f7aa4af9b3a-kube-api-access-4g5kc\") pod \"kas-connection-checker-685895c7c9-zwfbr\" (UID: \"239195cf-db90-4e49-9064-8f7aa4af9b3a\") " pod="kube-system/kas-connection-checker-685895c7c9-zwfbr" Oct 30 06:56:34.888351 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.888247 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-sysconfig\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.888351 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.888278 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-env-overrides\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.888351 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.888339 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-run-k8s-cni-cncf-io\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.889015 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.888367 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp7gc\" (UniqueName: \"kubernetes.io/projected/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-kube-api-access-gp7gc\") pod \"cluster-monitoring-operator-884576c57-l6qcv\" (UID: \"8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l6qcv" Oct 30 06:56:34.889015 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.888400 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:56:34.889015 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.888431 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-run-multus-certs\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.889015 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.888462 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/36b00a60-14fb-4364-8943-6bcacc5365cc-ca-trust-extracted\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:34.889015 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.888496 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/36b00a60-14fb-4364-8943-6bcacc5365cc-installation-pull-secrets\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:34.889015 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.888529 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-serving-cert\") pod \"insights-operator-7c4986d449-qqt8g\" (UID: \"4b59e668-d5a4-4ce0-a1e9-a16d63f51990\") " pod="openshift-insights/insights-operator-7c4986d449-qqt8g" Oct 30 06:56:34.889015 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.888560 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bbda0b6e-80fb-4964-a0a9-23c69a84889e-iptables-alerter-script\") pod \"iptables-alerter-zzvr7\" (UID: \"bbda0b6e-80fb-4964-a0a9-23c69a84889e\") " pod="openshift-network-operator/iptables-alerter-zzvr7" Oct 30 06:56:34.889015 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.888601 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-var-lib-kubelet\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.889015 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.888634 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm5vg\" (UniqueName: \"kubernetes.io/projected/c5f1f83f-9bfd-45f5-b065-ba95cf659956-kube-api-access-qm5vg\") pod \"service-ca-operator-544d749c8c-2kt77\" (UID: \"c5f1f83f-9bfd-45f5-b065-ba95cf659956\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2kt77" Oct 30 06:56:34.889015 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.888666 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9d1faf4-274f-49c8-9340-f85463840b27-image-registry-private-configuration\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:34.889015 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.888696 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d1b84f62-7f09-42d0-8ed1-a2447efa3bf3-host\") pod \"node-ca-9fpvq\" (UID: \"d1b84f62-7f09-42d0-8ed1-a2447efa3bf3\") " pod="openshift-image-registry/node-ca-9fpvq" Oct 30 06:56:34.889015 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.888723 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbpxj\" (UniqueName: \"kubernetes.io/projected/42802760-36a8-4b79-8690-5a00528503c7-kube-api-access-zbpxj\") pod \"console-operator-7cd56b8dbc-zw4km\" (UID: \"42802760-36a8-4b79-8690-5a00528503c7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" Oct 30 06:56:34.889015 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.888760 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx6zb\" (UniqueName: \"kubernetes.io/projected/1ab9750b-0582-4c7c-b770-b1b7fbe8994c-kube-api-access-jx6zb\") pod \"ingress-canary-qrksf\" (UID: \"1ab9750b-0582-4c7c-b770-b1b7fbe8994c\") " pod="openshift-ingress-canary/ingress-canary-qrksf" Oct 30 06:56:34.889015 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.888790 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l5w2\" (UniqueName: \"kubernetes.io/projected/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-kube-api-access-2l5w2\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.889015 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.888820 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c5f1f83f-9bfd-45f5-b065-ba95cf659956-config\") pod \"service-ca-operator-544d749c8c-2kt77\" (UID: \"c5f1f83f-9bfd-45f5-b065-ba95cf659956\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2kt77" Oct 30 06:56:34.889015 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.888853 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jldj6\" (UniqueName: \"kubernetes.io/projected/1327b481-6353-4d2d-8847-caa7fc7e3a3d-kube-api-access-jldj6\") pod \"migrator-66df5d496b-6t2h6\" (UID: \"1327b481-6353-4d2d-8847-caa7fc7e3a3d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6t2h6" Oct 30 06:56:34.889015 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.888877 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-run\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.889015 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.888908 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-default-certificate\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:34.889015 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.888941 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/20a9a026-75f0-4e90-b66a-066d3b80facf-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-8xtng\" (UID: \"20a9a026-75f0-4e90-b66a-066d3b80facf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8xtng" Oct 30 06:56:34.889015 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.888971 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-run-ovn\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.889015 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.889006 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-cnibin\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.889994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.889031 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-etc-selinux\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:56:34.889994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.889062 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgbhv\" (UniqueName: \"kubernetes.io/projected/d930accb-e4d8-404b-8155-1b6a8edd812e-kube-api-access-lgbhv\") pod \"dns-default-vgrqz\" (UID: \"d930accb-e4d8-404b-8155-1b6a8edd812e\") " pod="openshift-dns/dns-default-vgrqz" Oct 30 06:56:34.889994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.889093 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-cnibin\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:56:34.889994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.889124 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67hxp\" (UniqueName: \"kubernetes.io/projected/20a9a026-75f0-4e90-b66a-066d3b80facf-kube-api-access-67hxp\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-8xtng\" (UID: \"20a9a026-75f0-4e90-b66a-066d3b80facf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8xtng" Oct 30 06:56:34.889994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.889149 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/42802760-36a8-4b79-8690-5a00528503c7-config\") pod \"console-operator-7cd56b8dbc-zw4km\" (UID: \"42802760-36a8-4b79-8690-5a00528503c7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" Oct 30 06:56:34.889994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.889179 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-bound-sa-token\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:34.889994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.889208 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-var-lib-cni-bin\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.889994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.889239 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v8q5\" (UniqueName: \"kubernetes.io/projected/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-kube-api-access-6v8q5\") pod \"network-metrics-daemon-g6vbv\" (UID: \"f1b14112-6c97-4bfa-8f2a-5988f59ad06a\") " pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:56:34.889994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.889298 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-kubernetes\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.889994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.889322 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-sysctl-d\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.889994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.889354 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/36b00a60-14fb-4364-8943-6bcacc5365cc-image-registry-private-configuration\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:34.889994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.889384 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-l6qcv\" (UID: \"8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l6qcv" Oct 30 06:56:34.889994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.889413 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ab9750b-0582-4c7c-b770-b1b7fbe8994c-cert\") pod \"ingress-canary-qrksf\" (UID: \"1ab9750b-0582-4c7c-b770-b1b7fbe8994c\") " pod="openshift-ingress-canary/ingress-canary-qrksf" Oct 30 06:56:34.889994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.889441 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-run-ovn-kubernetes\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.889994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.889465 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-crio-socket\") pod \"insights-runtime-extractor-r5bpp\" (UID: \"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3\") " pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:34.889994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.889496 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s88rb\" (UniqueName: \"kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-kube-api-access-s88rb\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:34.889994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.889527 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5943f8dc-3e86-49fa-ac40-70a1f11fb69e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-cjn9t\" (UID: \"5943f8dc-3e86-49fa-ac40-70a1f11fb69e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cjn9t" Oct 30 06:56:34.889994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.889586 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5943f8dc-3e86-49fa-ac40-70a1f11fb69e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cjn9t\" (UID: \"5943f8dc-3e86-49fa-ac40-70a1f11fb69e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cjn9t" Oct 30 06:56:34.889994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.889616 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbp46\" (UniqueName: \"kubernetes.io/projected/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-kube-api-access-xbp46\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.889994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.889646 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-stats-auth\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:34.889994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.889677 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-run-netns\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.889994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.889708 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-cni-bin\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.889994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.889733 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-cni-netd\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.889994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.889765 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-original-pull-secret\") pod \"global-pull-secret-syncer-ts77g\" (UID: \"96024683-dd4e-4d78-b5ad-6aa9ccc4bc05\") " pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:56:34.889994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.889796 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tf48\" (UniqueName: \"kubernetes.io/projected/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-kube-api-access-2tf48\") pod \"insights-runtime-extractor-r5bpp\" (UID: \"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3\") " pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:34.889994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.889828 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:56:34.889994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.889860 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/36b00a60-14fb-4364-8943-6bcacc5365cc-trusted-ca\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:34.889994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.889886 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-registration-dir\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:56:34.889994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.889916 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-metrics-certs\") pod \"network-metrics-daemon-g6vbv\" (UID: \"f1b14112-6c97-4bfa-8f2a-5988f59ad06a\") " pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:56:34.889994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.889945 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4664d551-2cbf-42c5-a730-0d8c088f4f0f-hosts-file\") pod \"node-resolver-qq6f5\" (UID: \"4664d551-2cbf-42c5-a730-0d8c088f4f0f\") " pod="openshift-dns/node-resolver-qq6f5" Oct 30 06:56:34.889994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.889977 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj7gf\" (UniqueName: \"kubernetes.io/projected/bbda0b6e-80fb-4964-a0a9-23c69a84889e-kube-api-access-zj7gf\") pod \"iptables-alerter-zzvr7\" (UID: \"bbda0b6e-80fb-4964-a0a9-23c69a84889e\") " pod="openshift-network-operator/iptables-alerter-zzvr7" Oct 30 06:56:34.889994 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.890007 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r5bpp\" (UID: \"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3\") " pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:34.891401 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.890035 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-cni-binary-copy\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:56:34.891401 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.890069 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:34.891401 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.890098 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-socket-dir\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:56:34.891401 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.890129 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-sys-fs\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:56:34.891401 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.890163 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-945vk\" (UniqueName: \"kubernetes.io/projected/35296121-39f4-4fae-ae2f-f92790ad3f5c-kube-api-access-945vk\") pod \"network-check-source-6f78d59f99-fv48w\" (UID: \"35296121-39f4-4fae-ae2f-f92790ad3f5c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fv48w" Oct 30 06:56:34.891401 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.890190 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-systemd\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.891401 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.890222 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-multus-cni-dir\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.891401 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.890254 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9d1faf4-274f-49c8-9340-f85463840b27-ca-trust-extracted\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:34.891401 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.890290 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-984ts\" (UniqueName: \"kubernetes.io/projected/584e4eb5-6e5e-4f96-b97a-eb58fd762347-kube-api-access-984ts\") pod \"volume-data-source-validator-58c9f5f46d-z5nm2\" (UID: \"584e4eb5-6e5e-4f96-b97a-eb58fd762347\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z5nm2" Oct 30 06:56:34.891401 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.890318 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-tuned\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.891401 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.890349 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-run-openvswitch\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.891401 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.890381 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/42802760-36a8-4b79-8690-5a00528503c7-serving-cert\") pod \"console-operator-7cd56b8dbc-zw4km\" (UID: \"42802760-36a8-4b79-8690-5a00528503c7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" Oct 30 06:56:34.891401 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.890413 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-etc-openvswitch\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.891401 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.890441 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-node-log\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.891401 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.890467 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d930accb-e4d8-404b-8155-1b6a8edd812e-metrics-tls\") pod \"dns-default-vgrqz\" (UID: \"d930accb-e4d8-404b-8155-1b6a8edd812e\") " pod="openshift-dns/dns-default-vgrqz" Oct 30 06:56:34.891401 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.890499 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9d1faf4-274f-49c8-9340-f85463840b27-installation-pull-secrets\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:34.891401 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.890530 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-modprobe-d\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.891401 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.890564 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.891401 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.890613 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-ovnkube-script-lib\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.891401 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.890641 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-dbus\") pod \"global-pull-secret-syncer-ts77g\" (UID: \"96024683-dd4e-4d78-b5ad-6aa9ccc4bc05\") " pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:56:34.891401 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.890673 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d930accb-e4d8-404b-8155-1b6a8edd812e-tmp-dir\") pod \"dns-default-vgrqz\" (UID: \"d930accb-e4d8-404b-8155-1b6a8edd812e\") " pod="openshift-dns/dns-default-vgrqz" Oct 30 06:56:34.891401 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.890705 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-multus-socket-dir-parent\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.891401 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.890740 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-var-lib-cni-multus\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.891401 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.890768 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9d1faf4-274f-49c8-9340-f85463840b27-registry-certificates\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:34.891401 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.890800 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-sys\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.891401 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.890830 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-tmp\") pod \"insights-operator-7c4986d449-qqt8g\" (UID: \"4b59e668-d5a4-4ce0-a1e9-a16d63f51990\") " pod="openshift-insights/insights-operator-7c4986d449-qqt8g" Oct 30 06:56:34.891401 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.890949 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-tmp\") pod \"insights-operator-7c4986d449-qqt8g\" (UID: \"4b59e668-d5a4-4ce0-a1e9-a16d63f51990\") " pod="openshift-insights/insights-operator-7c4986d449-qqt8g" Oct 30 06:56:34.891401 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.891107 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c5f1f83f-9bfd-45f5-b065-ba95cf659956-serving-cert\") pod \"service-ca-operator-544d749c8c-2kt77\" (UID: \"c5f1f83f-9bfd-45f5-b065-ba95cf659956\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2kt77" Oct 30 06:56:34.892508 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.891933 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/36b00a60-14fb-4364-8943-6bcacc5365cc-installation-pull-secrets\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:34.892508 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.891970 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/42802760-36a8-4b79-8690-5a00528503c7-serving-cert\") pod \"console-operator-7cd56b8dbc-zw4km\" (UID: \"42802760-36a8-4b79-8690-5a00528503c7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" Oct 30 06:56:34.892508 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.892104 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/20a9a026-75f0-4e90-b66a-066d3b80facf-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-8xtng\" (UID: \"20a9a026-75f0-4e90-b66a-066d3b80facf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8xtng" Oct 30 06:56:34.892508 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.892205 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-cni-binary-copy\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.892508 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.892219 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ab9750b-0582-4c7c-b770-b1b7fbe8994c-cert\") pod \"ingress-canary-qrksf\" (UID: \"1ab9750b-0582-4c7c-b770-b1b7fbe8994c\") " pod="openshift-ingress-canary/ingress-canary-qrksf" Oct 30 06:56:34.892508 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.892258 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:56:34.892508 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.892328 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9d1faf4-274f-49c8-9340-f85463840b27-ca-trust-extracted\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:34.892818 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.892331 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-service-ca-bundle\") pod \"insights-operator-7c4986d449-qqt8g\" (UID: \"4b59e668-d5a4-4ce0-a1e9-a16d63f51990\") " pod="openshift-insights/insights-operator-7c4986d449-qqt8g" Oct 30 06:56:34.892818 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.892641 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-original-pull-secret\") pod \"global-pull-secret-syncer-ts77g\" (UID: \"96024683-dd4e-4d78-b5ad-6aa9ccc4bc05\") " pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:56:34.892818 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.892712 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9d1faf4-274f-49c8-9340-f85463840b27-image-registry-private-configuration\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:34.892954 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.892822 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5225f97b-21b7-453c-8d19-ccfce5ee6d54-agent-certs\") pod \"konnectivity-agent-mjwrd\" (UID: \"5225f97b-21b7-453c-8d19-ccfce5ee6d54\") " pod="kube-system/konnectivity-agent-mjwrd" Oct 30 06:56:34.892954 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.892931 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d930accb-e4d8-404b-8155-1b6a8edd812e-tmp-dir\") pod \"dns-default-vgrqz\" (UID: \"d930accb-e4d8-404b-8155-1b6a8edd812e\") " pod="openshift-dns/dns-default-vgrqz" Oct 30 06:56:34.893066 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.892992 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9d1faf4-274f-49c8-9340-f85463840b27-installation-pull-secrets\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:34.893369 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.893122 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5943f8dc-3e86-49fa-ac40-70a1f11fb69e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-cjn9t\" (UID: \"5943f8dc-3e86-49fa-ac40-70a1f11fb69e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cjn9t" Oct 30 06:56:34.893369 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.893193 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/36b00a60-14fb-4364-8943-6bcacc5365cc-ca-trust-extracted\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:34.893369 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.893204 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:56:34.893369 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.893313 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-ovnkube-script-lib\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.893369 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.893315 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/36b00a60-14fb-4364-8943-6bcacc5365cc-trusted-ca\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:34.893659 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.893404 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-l6qcv\" (UID: \"8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l6qcv" Oct 30 06:56:34.893659 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.893419 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/42802760-36a8-4b79-8690-5a00528503c7-config\") pod \"console-operator-7cd56b8dbc-zw4km\" (UID: \"42802760-36a8-4b79-8690-5a00528503c7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" Oct 30 06:56:34.893659 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.893436 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bbda0b6e-80fb-4964-a0a9-23c69a84889e-iptables-alerter-script\") pod \"iptables-alerter-zzvr7\" (UID: \"bbda0b6e-80fb-4964-a0a9-23c69a84889e\") " pod="openshift-network-operator/iptables-alerter-zzvr7" Oct 30 06:56:34.893659 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.893523 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/36b00a60-14fb-4364-8943-6bcacc5365cc-image-registry-private-configuration\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:34.893659 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.893524 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-env-overrides\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.893659 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.893553 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-serving-cert\") pod \"insights-operator-7c4986d449-qqt8g\" (UID: \"4b59e668-d5a4-4ce0-a1e9-a16d63f51990\") " pod="openshift-insights/insights-operator-7c4986d449-qqt8g" Oct 30 06:56:34.893932 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.893694 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-cni-binary-copy\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:56:34.893932 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.893724 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-stats-auth\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:34.893932 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.893791 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-tuned\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.893932 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.893884 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9d1faf4-274f-49c8-9340-f85463840b27-registry-certificates\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:34.893932 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.893899 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-default-certificate\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:34.894114 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.893976 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c5f1f83f-9bfd-45f5-b065-ba95cf659956-config\") pod \"service-ca-operator-544d749c8c-2kt77\" (UID: \"c5f1f83f-9bfd-45f5-b065-ba95cf659956\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2kt77" Oct 30 06:56:34.912778 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.912752 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9jzs\" (UniqueName: \"kubernetes.io/projected/0994513f-0dac-4106-8bfe-5a89cbc11f02-kube-api-access-x9jzs\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.912873 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.912776 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-bound-sa-token\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:34.912873 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.912802 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6527\" (UniqueName: \"kubernetes.io/projected/4664d551-2cbf-42c5-a730-0d8c088f4f0f-kube-api-access-t6527\") pod \"node-resolver-qq6f5\" (UID: \"4664d551-2cbf-42c5-a730-0d8c088f4f0f\") " pod="openshift-dns/node-resolver-qq6f5" Oct 30 06:56:34.913002 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.912905 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlg4z\" (UniqueName: \"kubernetes.io/projected/d1b84f62-7f09-42d0-8ed1-a2447efa3bf3-kube-api-access-tlg4z\") pod \"node-ca-9fpvq\" (UID: \"d1b84f62-7f09-42d0-8ed1-a2447efa3bf3\") " pod="openshift-image-registry/node-ca-9fpvq" Oct 30 06:56:34.913067 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.913047 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrc6h\" (UniqueName: \"kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-kube-api-access-rrc6h\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:34.913525 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.913509 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9n88\" (UniqueName: \"kubernetes.io/projected/4b59e668-d5a4-4ce0-a1e9-a16d63f51990-kube-api-access-z9n88\") pod \"insights-operator-7c4986d449-qqt8g\" (UID: \"4b59e668-d5a4-4ce0-a1e9-a16d63f51990\") " pod="openshift-insights/insights-operator-7c4986d449-qqt8g" Oct 30 06:56:34.913689 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.913675 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vrh9\" (UniqueName: \"kubernetes.io/projected/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-kube-api-access-4vrh9\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:34.914518 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.914503 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6q49\" (UniqueName: \"kubernetes.io/projected/14b28600-2ad3-47ff-a157-bcfacc6ab2ff-kube-api-access-z6q49\") pod \"kas-connection-checker-685895c7c9-7dwjw\" (UID: \"14b28600-2ad3-47ff-a157-bcfacc6ab2ff\") " pod="kube-system/kas-connection-checker-685895c7c9-7dwjw" Oct 30 06:56:34.920034 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.920013 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tf48\" (UniqueName: \"kubernetes.io/projected/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-kube-api-access-2tf48\") pod \"insights-runtime-extractor-r5bpp\" (UID: \"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3\") " pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:34.926008 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.925987 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s88rb\" (UniqueName: \"kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-kube-api-access-s88rb\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:34.926663 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.926644 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbp46\" (UniqueName: \"kubernetes.io/projected/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-kube-api-access-xbp46\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.927556 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.927540 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h86bq\" (UniqueName: \"kubernetes.io/projected/50ab5bd7-ea08-460b-9932-1dac54f3005b-kube-api-access-h86bq\") pod \"cluster-samples-operator-6ddb6c8b94-cflrf\" (UID: \"50ab5bd7-ea08-460b-9932-1dac54f3005b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cflrf" Oct 30 06:56:34.928080 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.928061 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-945vk\" (UniqueName: \"kubernetes.io/projected/35296121-39f4-4fae-ae2f-f92790ad3f5c-kube-api-access-945vk\") pod \"network-check-source-6f78d59f99-fv48w\" (UID: \"35296121-39f4-4fae-ae2f-f92790ad3f5c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fv48w" Oct 30 06:56:34.928534 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.928519 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp7gc\" (UniqueName: \"kubernetes.io/projected/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-kube-api-access-gp7gc\") pod \"cluster-monitoring-operator-884576c57-l6qcv\" (UID: \"8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l6qcv" Oct 30 06:56:34.930530 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.930510 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g5kc\" (UniqueName: \"kubernetes.io/projected/239195cf-db90-4e49-9064-8f7aa4af9b3a-kube-api-access-4g5kc\") pod \"kas-connection-checker-685895c7c9-zwfbr\" (UID: \"239195cf-db90-4e49-9064-8f7aa4af9b3a\") " pod="kube-system/kas-connection-checker-685895c7c9-zwfbr" Oct 30 06:56:34.930628 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.930613 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jldj6\" (UniqueName: \"kubernetes.io/projected/1327b481-6353-4d2d-8847-caa7fc7e3a3d-kube-api-access-jldj6\") pod \"migrator-66df5d496b-6t2h6\" (UID: \"1327b481-6353-4d2d-8847-caa7fc7e3a3d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6t2h6" Oct 30 06:56:34.933445 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.933425 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bth9\" (UniqueName: \"kubernetes.io/projected/8f93e69e-f48e-4c0a-ab3d-655739b6a8ae-kube-api-access-4bth9\") pod \"kas-connection-checker-685895c7c9-5bl7s\" (UID: \"8f93e69e-f48e-4c0a-ab3d-655739b6a8ae\") " pod="kube-system/kas-connection-checker-685895c7c9-5bl7s" Oct 30 06:56:34.934282 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.934267 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-984ts\" (UniqueName: \"kubernetes.io/projected/584e4eb5-6e5e-4f96-b97a-eb58fd762347-kube-api-access-984ts\") pod \"volume-data-source-validator-58c9f5f46d-z5nm2\" (UID: \"584e4eb5-6e5e-4f96-b97a-eb58fd762347\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z5nm2" Oct 30 06:56:34.940386 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.940366 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm5vg\" (UniqueName: \"kubernetes.io/projected/c5f1f83f-9bfd-45f5-b065-ba95cf659956-kube-api-access-qm5vg\") pod \"service-ca-operator-544d749c8c-2kt77\" (UID: \"c5f1f83f-9bfd-45f5-b065-ba95cf659956\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2kt77" Oct 30 06:56:34.940494 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.940473 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l5w2\" (UniqueName: \"kubernetes.io/projected/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-kube-api-access-2l5w2\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.940542 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.940488 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj7gf\" (UniqueName: \"kubernetes.io/projected/bbda0b6e-80fb-4964-a0a9-23c69a84889e-kube-api-access-zj7gf\") pod \"iptables-alerter-zzvr7\" (UID: \"bbda0b6e-80fb-4964-a0a9-23c69a84889e\") " pod="openshift-network-operator/iptables-alerter-zzvr7" Oct 30 06:56:34.941317 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.941293 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-bound-sa-token\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:34.941412 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.941368 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v8q5\" (UniqueName: \"kubernetes.io/projected/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-kube-api-access-6v8q5\") pod \"network-metrics-daemon-g6vbv\" (UID: \"f1b14112-6c97-4bfa-8f2a-5988f59ad06a\") " pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:56:34.941412 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.941379 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7ndt\" (UniqueName: \"kubernetes.io/projected/16e1a28b-a156-4d4e-820f-8712616b4eb2-kube-api-access-x7ndt\") pod \"network-check-target-rjdmc\" (UID: \"16e1a28b-a156-4d4e-820f-8712616b4eb2\") " pod="openshift-network-diagnostics/network-check-target-rjdmc" Oct 30 06:56:34.941531 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.941456 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbpxj\" (UniqueName: \"kubernetes.io/projected/42802760-36a8-4b79-8690-5a00528503c7-kube-api-access-zbpxj\") pod \"console-operator-7cd56b8dbc-zw4km\" (UID: \"42802760-36a8-4b79-8690-5a00528503c7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" Oct 30 06:56:34.941531 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.941460 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgbhv\" (UniqueName: \"kubernetes.io/projected/d930accb-e4d8-404b-8155-1b6a8edd812e-kube-api-access-lgbhv\") pod \"dns-default-vgrqz\" (UID: \"d930accb-e4d8-404b-8155-1b6a8edd812e\") " pod="openshift-dns/dns-default-vgrqz" Oct 30 06:56:34.941841 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.941824 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx6zb\" (UniqueName: \"kubernetes.io/projected/1ab9750b-0582-4c7c-b770-b1b7fbe8994c-kube-api-access-jx6zb\") pod \"ingress-canary-qrksf\" (UID: \"1ab9750b-0582-4c7c-b770-b1b7fbe8994c\") " pod="openshift-ingress-canary/ingress-canary-qrksf" Oct 30 06:56:34.943319 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.943299 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bx54\" (UniqueName: \"kubernetes.io/projected/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-kube-api-access-2bx54\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:56:34.944005 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.943984 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbzjd\" (UniqueName: \"kubernetes.io/projected/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-kube-api-access-fbzjd\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:56:34.947176 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.947157 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67hxp\" (UniqueName: \"kubernetes.io/projected/20a9a026-75f0-4e90-b66a-066d3b80facf-kube-api-access-67hxp\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-8xtng\" (UID: \"20a9a026-75f0-4e90-b66a-066d3b80facf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8xtng" Oct 30 06:56:34.991368 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991283 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-var-lib-kubelet\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.991368 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991331 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-kubelet\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.991536 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991418 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-var-lib-kubelet\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.991536 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991437 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-device-dir\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:56:34.991536 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991467 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-kubelet-config\") pod \"global-pull-secret-syncer-ts77g\" (UID: \"96024683-dd4e-4d78-b5ad-6aa9ccc4bc05\") " pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:56:34.991536 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991469 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-kubelet\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.991536 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991489 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-hostroot\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.991536 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991495 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-device-dir\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:56:34.991536 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991507 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:56:34.991536 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991522 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-hostroot\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.991536 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991527 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-sysconfig\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.991831 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991545 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-run-k8s-cni-cncf-io\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.991831 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991547 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-kubelet-config\") pod \"global-pull-secret-syncer-ts77g\" (UID: \"96024683-dd4e-4d78-b5ad-6aa9ccc4bc05\") " pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:56:34.991831 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991549 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:56:34.991831 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991589 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-sysconfig\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.991831 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991561 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-run-multus-certs\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.991831 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991606 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-run-k8s-cni-cncf-io\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.991831 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991622 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-run-multus-certs\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.991831 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991626 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-var-lib-kubelet\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.991831 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991651 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-var-lib-kubelet\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.991831 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991659 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d1b84f62-7f09-42d0-8ed1-a2447efa3bf3-host\") pod \"node-ca-9fpvq\" (UID: \"d1b84f62-7f09-42d0-8ed1-a2447efa3bf3\") " pod="openshift-image-registry/node-ca-9fpvq" Oct 30 06:56:34.991831 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991682 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-run\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.991831 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991697 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-run-ovn\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.991831 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991715 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-cnibin\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.991831 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991739 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-etc-selinux\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:56:34.991831 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991765 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-cnibin\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:56:34.991831 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991741 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-run-ovn\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.991831 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991776 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-etc-selinux\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:56:34.991831 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991778 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-cnibin\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.991831 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991791 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-var-lib-cni-bin\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.991831 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991797 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-cnibin\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:56:34.991831 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991764 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d1b84f62-7f09-42d0-8ed1-a2447efa3bf3-host\") pod \"node-ca-9fpvq\" (UID: \"d1b84f62-7f09-42d0-8ed1-a2447efa3bf3\") " pod="openshift-image-registry/node-ca-9fpvq" Oct 30 06:56:34.991831 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991762 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-run\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.991831 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991814 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-kubernetes\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.991831 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991840 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-sysctl-d\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991845 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-var-lib-cni-bin\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991859 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-kubernetes\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991885 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-run-ovn-kubernetes\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991900 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-run-ovn-kubernetes\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991913 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-crio-socket\") pod \"insights-runtime-extractor-r5bpp\" (UID: \"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3\") " pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991932 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-sysctl-d\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.991936 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5943f8dc-3e86-49fa-ac40-70a1f11fb69e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cjn9t\" (UID: \"5943f8dc-3e86-49fa-ac40-70a1f11fb69e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cjn9t" Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992007 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-run-netns\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992029 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-crio-socket\") pod \"insights-runtime-extractor-r5bpp\" (UID: \"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3\") " pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992035 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-cni-bin\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992061 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-cni-netd\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992068 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-run-netns\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992089 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-cni-bin\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992088 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992105 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-cni-netd\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992114 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-registration-dir\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992130 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992136 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-metrics-certs\") pod \"network-metrics-daemon-g6vbv\" (UID: \"f1b14112-6c97-4bfa-8f2a-5988f59ad06a\") " pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992170 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-registration-dir\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992193 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4664d551-2cbf-42c5-a730-0d8c088f4f0f-hosts-file\") pod \"node-resolver-qq6f5\" (UID: \"4664d551-2cbf-42c5-a730-0d8c088f4f0f\") " pod="openshift-dns/node-resolver-qq6f5" Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:34.992217 4951 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992252 4951 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:34.992272 4951 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992273 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4664d551-2cbf-42c5-a730-0d8c088f4f0f-hosts-file\") pod \"node-resolver-qq6f5\" (UID: \"4664d551-2cbf-42c5-a730-0d8c088f4f0f\") " pod="openshift-dns/node-resolver-qq6f5" Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992219 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r5bpp\" (UID: \"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3\") " pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:34.992318 4951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-metrics-certs podName:f1b14112-6c97-4bfa-8f2a-5988f59ad06a nodeName:}" failed. No retries permitted until 2026-10-30 06:56:35.492285173 +0000 UTC m=+1.835333133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-metrics-certs") pod "network-metrics-daemon-g6vbv" (UID: "f1b14112-6c97-4bfa-8f2a-5988f59ad06a") : secret "metrics-daemon-secret" not found Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:34.992377 4951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-insights-runtime-extractor-tls podName:4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:35.492355362 +0000 UTC m=+1.835403321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r5bpp" (UID: "4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3") : secret "insights-runtime-extractor-tls" not found Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992400 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992425 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-socket-dir\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992445 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-sys-fs\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:34.992459 4951 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:34.992469 4951 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dcb9bff4d-5xf9f: secret "image-registry-tls" not found Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992493 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-systemd\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992459 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-systemd\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:34.992524 4951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls podName:36b00a60-14fb-4364-8943-6bcacc5365cc nodeName:}" failed. No retries permitted until 2026-10-30 06:56:35.492509592 +0000 UTC m=+1.835557561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls") pod "image-registry-5dcb9bff4d-5xf9f" (UID: "36b00a60-14fb-4364-8943-6bcacc5365cc") : secret "image-registry-tls" not found Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992538 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-socket-dir\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992550 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-multus-cni-dir\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992542 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a4aa6b79-3b44-4b45-834c-78ee0fb91e02-sys-fs\") pod \"aws-ebs-csi-driver-node-9lgv4\" (UID: \"a4aa6b79-3b44-4b45-834c-78ee0fb91e02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9lgv4" Oct 30 06:56:34.992584 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992597 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-run-openvswitch\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992628 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-etc-openvswitch\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992633 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-run-openvswitch\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992652 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-node-log\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992672 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-etc-openvswitch\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992630 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-multus-cni-dir\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992678 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d930accb-e4d8-404b-8155-1b6a8edd812e-metrics-tls\") pod \"dns-default-vgrqz\" (UID: \"d930accb-e4d8-404b-8155-1b6a8edd812e\") " pod="openshift-dns/dns-default-vgrqz" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992709 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-node-log\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992717 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-modprobe-d\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:34.992736 4951 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:34.992763 4951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d930accb-e4d8-404b-8155-1b6a8edd812e-metrics-tls podName:d930accb-e4d8-404b-8155-1b6a8edd812e nodeName:}" failed. No retries permitted until 2026-10-30 06:56:35.492755796 +0000 UTC m=+1.835803739 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d930accb-e4d8-404b-8155-1b6a8edd812e-metrics-tls") pod "dns-default-vgrqz" (UID: "d930accb-e4d8-404b-8155-1b6a8edd812e") : secret "dns-default-metrics-tls" not found Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992761 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992737 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992789 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-dbus\") pod \"global-pull-secret-syncer-ts77g\" (UID: \"96024683-dd4e-4d78-b5ad-6aa9ccc4bc05\") " pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992815 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-multus-socket-dir-parent\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992840 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-var-lib-cni-multus\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992840 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96024683-dd4e-4d78-b5ad-6aa9ccc4bc05-dbus\") pod \"global-pull-secret-syncer-ts77g\" (UID: \"96024683-dd4e-4d78-b5ad-6aa9ccc4bc05\") " pod="kube-system/global-pull-secret-syncer-ts77g" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992792 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-modprobe-d\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992868 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-sys\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992879 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-multus-socket-dir-parent\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992889 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-var-lib-cni-multus\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992892 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-multus-conf-dir\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992905 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-sys\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992915 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-var-lib-openvswitch\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992936 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-multus-conf-dir\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992940 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-var-lib-openvswitch\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992956 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-run-netns\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.992983 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-slash\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.993004 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-etc-kubernetes\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.993024 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.993042 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-host-run-netns\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.993046 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-host-slash\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.993052 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-etc-kubernetes\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.993053 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-service-ca-bundle\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.993094 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-systemd-units\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:34.993110 4951 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:34.993116 4951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-service-ca-bundle podName:0c49a4a4-033f-4a62-8bc4-6b59531e8bf7 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:35.493105944 +0000 UTC m=+1.836153917 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-service-ca-bundle") pod "router-default-7f6bb8cb6f-fzpwq" (UID: "0c49a4a4-033f-4a62-8bc4-6b59531e8bf7") : configmap references non-existent config key: service-ca.crt Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:34.993121 4951 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b5f74d469-87gxs: secret "image-registry-tls" not found Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.993135 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-system-cni-dir\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.993134 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-systemd-units\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:34.993154 4951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls podName:b9d1faf4-274f-49c8-9340-f85463840b27 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:35.49314259 +0000 UTC m=+1.836190551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls") pod "image-registry-5b5f74d469-87gxs" (UID: "b9d1faf4-274f-49c8-9340-f85463840b27") : secret "image-registry-tls" not found Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.993166 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/044f4f8f-15bd-4f21-b6ca-2afa1f1171c5-system-cni-dir\") pod \"multus-additional-cni-plugins-t4rnr\" (UID: \"044f4f8f-15bd-4f21-b6ca-2afa1f1171c5\") " pod="openshift-multus/multus-additional-cni-plugins-t4rnr" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.993177 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-lib-modules\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.993195 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l6qcv\" (UID: \"8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l6qcv" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.993227 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-host\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.993252 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bbda0b6e-80fb-4964-a0a9-23c69a84889e-host-slash\") pod \"iptables-alerter-zzvr7\" (UID: \"bbda0b6e-80fb-4964-a0a9-23c69a84889e\") " pod="openshift-network-operator/iptables-alerter-zzvr7" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.993267 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-lib-modules\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.993279 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-sysctl-conf\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.993292 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-host\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.993297 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bbda0b6e-80fb-4964-a0a9-23c69a84889e-host-slash\") pod \"iptables-alerter-zzvr7\" (UID: \"bbda0b6e-80fb-4964-a0a9-23c69a84889e\") " pod="openshift-network-operator/iptables-alerter-zzvr7" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:34.993256 4951 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.993308 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-metrics-certs\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.993337 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r5bpp\" (UID: \"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3\") " pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:34.993352 4951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-cluster-monitoring-operator-tls podName:8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:35.493338496 +0000 UTC m=+1.836386453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-l6qcv" (UID: "8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2") : secret "cluster-monitoring-operator-tls" not found Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.993372 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0994513f-0dac-4106-8bfe-5a89cbc11f02-etc-sysctl-conf\") pod \"tuned-qqnmt\" (UID: \"0994513f-0dac-4106-8bfe-5a89cbc11f02\") " pod="openshift-cluster-node-tuning-operator/tuned-qqnmt" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.993378 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/50ab5bd7-ea08-460b-9932-1dac54f3005b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-cflrf\" (UID: \"50ab5bd7-ea08-460b-9932-1dac54f3005b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cflrf" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:34.993409 4951 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.993412 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-run-systemd\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.993436 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-log-socket\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.993453 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-run-systemd\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:34.993464 4951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-metrics-certs podName:0c49a4a4-033f-4a62-8bc4-6b59531e8bf7 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:35.493452574 +0000 UTC m=+1.836500518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-metrics-certs") pod "router-default-7f6bb8cb6f-fzpwq" (UID: "0c49a4a4-033f-4a62-8bc4-6b59531e8bf7") : secret "router-metrics-certs-default" not found Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.993476 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bd315a45-3a31-44f2-9bf1-6d43df8e0c44-log-socket\") pod \"ovnkube-node-8tlbb\" (UID: \"bd315a45-3a31-44f2-9bf1-6d43df8e0c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.993506 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-system-cni-dir\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.993560 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3be0bd9d-9b78-40d8-bfcc-d0a591c108ce-system-cni-dir\") pod \"multus-q9jl6\" (UID: \"3be0bd9d-9b78-40d8-bfcc-d0a591c108ce\") " pod="openshift-multus/multus-q9jl6" Oct 30 06:56:34.994367 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.993800 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r5bpp\" (UID: \"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3\") " pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:34.996263 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.994975 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5943f8dc-3e86-49fa-ac40-70a1f11fb69e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cjn9t\" (UID: \"5943f8dc-3e86-49fa-ac40-70a1f11fb69e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cjn9t" Oct 30 06:56:34.996263 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:34.995646 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/50ab5bd7-ea08-460b-9932-1dac54f3005b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-cflrf\" (UID: \"50ab5bd7-ea08-460b-9932-1dac54f3005b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cflrf" Oct 30 06:56:35.092450 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:35.092418 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cflrf" Oct 30 06:56:35.093769 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:35.093746 4951 scope.go:122] "RemoveContainer" containerID="ab6a3aae9beab6299cb2ae4ffc208236cde7fa979fe642aba021008f4c9e948f" Oct 30 06:56:35.093851 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:35.093791 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-cjn9t" Oct 30 06:56:35.438280 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:35.438247 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cflrf"] Oct 30 06:56:35.460463 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:35.460436 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-cjn9t"] Oct 30 06:56:35.462872 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:35.462758 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5943f8dc_3e86_49fa_ac40_70a1f11fb69e.slice/crio-3bef8a979fcdd35871d6bc0e349a4f324faa57ab833ad6b4912bd3a8eca8d6e3 WatchSource:0}: Error finding container 3bef8a979fcdd35871d6bc0e349a4f324faa57ab833ad6b4912bd3a8eca8d6e3: Status 404 returned error can't find the container with id 3bef8a979fcdd35871d6bc0e349a4f324faa57ab833ad6b4912bd3a8eca8d6e3 Oct 30 06:56:35.465094 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:35.465079 4951 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:56:35.499634 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:35.499610 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-metrics-certs\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:35.499730 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:35.499671 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-metrics-certs\") pod \"network-metrics-daemon-g6vbv\" (UID: \"f1b14112-6c97-4bfa-8f2a-5988f59ad06a\") " pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:56:35.499730 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:35.499691 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r5bpp\" (UID: \"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3\") " pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:35.499730 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:35.499707 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:35.499884 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:35.499730 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d930accb-e4d8-404b-8155-1b6a8edd812e-metrics-tls\") pod \"dns-default-vgrqz\" (UID: \"d930accb-e4d8-404b-8155-1b6a8edd812e\") " pod="openshift-dns/dns-default-vgrqz" Oct 30 06:56:35.499884 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:35.499776 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:35.499884 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:35.499808 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-service-ca-bundle\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:35.499884 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:35.499833 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l6qcv\" (UID: \"8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l6qcv" Oct 30 06:56:35.499884 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:35.499809 4951 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 06:56:35.500068 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:35.499839 4951 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 06:56:35.500068 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:35.499921 4951 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dcb9bff4d-5xf9f: secret "image-registry-tls" not found Oct 30 06:56:35.500068 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:35.499931 4951 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 06:56:35.500068 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:35.499849 4951 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 06:56:35.500068 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:35.499851 4951 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 06:56:35.500068 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:35.499908 4951 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 06:56:35.500068 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:35.500006 4951 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b5f74d469-87gxs: secret "image-registry-tls" not found Oct 30 06:56:35.500068 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:35.499912 4951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d930accb-e4d8-404b-8155-1b6a8edd812e-metrics-tls podName:d930accb-e4d8-404b-8155-1b6a8edd812e nodeName:}" failed. No retries permitted until 2026-10-30 06:56:36.499893743 +0000 UTC m=+2.842941705 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d930accb-e4d8-404b-8155-1b6a8edd812e-metrics-tls") pod "dns-default-vgrqz" (UID: "d930accb-e4d8-404b-8155-1b6a8edd812e") : secret "dns-default-metrics-tls" not found Oct 30 06:56:35.500068 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:35.500057 4951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-service-ca-bundle podName:0c49a4a4-033f-4a62-8bc4-6b59531e8bf7 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:36.500026914 +0000 UTC m=+2.843074862 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-service-ca-bundle") pod "router-default-7f6bb8cb6f-fzpwq" (UID: "0c49a4a4-033f-4a62-8bc4-6b59531e8bf7") : configmap references non-existent config key: service-ca.crt Oct 30 06:56:35.500382 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:35.500084 4951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls podName:36b00a60-14fb-4364-8943-6bcacc5365cc nodeName:}" failed. No retries permitted until 2026-10-30 06:56:36.500073368 +0000 UTC m=+2.843121316 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls") pod "image-registry-5dcb9bff4d-5xf9f" (UID: "36b00a60-14fb-4364-8943-6bcacc5365cc") : secret "image-registry-tls" not found Oct 30 06:56:35.500382 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:35.500111 4951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-cluster-monitoring-operator-tls podName:8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:36.500097534 +0000 UTC m=+2.843145482 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-l6qcv" (UID: "8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2") : secret "cluster-monitoring-operator-tls" not found Oct 30 06:56:35.500382 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:35.500131 4951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-insights-runtime-extractor-tls podName:4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:36.500123249 +0000 UTC m=+2.843171199 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r5bpp" (UID: "4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3") : secret "insights-runtime-extractor-tls" not found Oct 30 06:56:35.500382 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:35.500153 4951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-metrics-certs podName:f1b14112-6c97-4bfa-8f2a-5988f59ad06a nodeName:}" failed. No retries permitted until 2026-10-30 06:56:36.500144351 +0000 UTC m=+2.843192310 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-metrics-certs") pod "network-metrics-daemon-g6vbv" (UID: "f1b14112-6c97-4bfa-8f2a-5988f59ad06a") : secret "metrics-daemon-secret" not found Oct 30 06:56:35.500382 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:35.500171 4951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls podName:b9d1faf4-274f-49c8-9340-f85463840b27 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:36.500163647 +0000 UTC m=+2.843211646 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls") pod "image-registry-5b5f74d469-87gxs" (UID: "b9d1faf4-274f-49c8-9340-f85463840b27") : secret "image-registry-tls" not found Oct 30 06:56:35.501832 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:35.501813 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-metrics-certs\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:35.827364 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:35.827284 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-cjn9t" event={"ID":"5943f8dc-3e86-49fa-ac40-70a1f11fb69e","Type":"ContainerStarted","Data":"3bef8a979fcdd35871d6bc0e349a4f324faa57ab833ad6b4912bd3a8eca8d6e3"} Oct 30 06:56:35.828637 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:35.828603 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cflrf" event={"ID":"50ab5bd7-ea08-460b-9932-1dac54f3005b","Type":"ContainerStarted","Data":"ef608c802f4dc21719a0cca975d85e382f19f2a7e0606b2623e6dc1c84bc19c8"} Oct 30 06:56:35.830212 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:35.830194 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/2.log" Oct 30 06:56:35.830665 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:35.830641 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/1.log" Oct 30 06:56:35.830733 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:35.830676 4951 generic.go:356] "Generic (PLEG): container finished" podID="42802760-36a8-4b79-8690-5a00528503c7" containerID="6c0a7f3d03d409278e136b4be6b4fd9b9861fc108c08e90429cb106204060163" exitCode=255 Oct 30 06:56:35.830778 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:35.830747 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" event={"ID":"42802760-36a8-4b79-8690-5a00528503c7","Type":"ContainerDied","Data":"6c0a7f3d03d409278e136b4be6b4fd9b9861fc108c08e90429cb106204060163"} Oct 30 06:56:35.830812 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:35.830791 4951 scope.go:122] "RemoveContainer" containerID="ab6a3aae9beab6299cb2ae4ffc208236cde7fa979fe642aba021008f4c9e948f" Oct 30 06:56:35.831199 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:35.831177 4951 scope.go:122] "RemoveContainer" containerID="6c0a7f3d03d409278e136b4be6b4fd9b9861fc108c08e90429cb106204060163" Oct 30 06:56:35.831394 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:35.831371 4951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zw4km_openshift-console-operator(42802760-36a8-4b79-8690-5a00528503c7)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" podUID="42802760-36a8-4b79-8690-5a00528503c7" Oct 30 06:56:35.832135 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:35.832119 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:36.293139 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:36.293106 4951 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" Oct 30 06:56:36.509585 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:36.509542 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-metrics-certs\") pod \"network-metrics-daemon-g6vbv\" (UID: \"f1b14112-6c97-4bfa-8f2a-5988f59ad06a\") " pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:56:36.509775 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:36.509625 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r5bpp\" (UID: \"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3\") " pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:36.509775 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:36.509658 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:36.509775 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:36.509695 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d930accb-e4d8-404b-8155-1b6a8edd812e-metrics-tls\") pod \"dns-default-vgrqz\" (UID: \"d930accb-e4d8-404b-8155-1b6a8edd812e\") " pod="openshift-dns/dns-default-vgrqz" Oct 30 06:56:36.509775 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:36.509738 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:36.509775 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:36.509764 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-service-ca-bundle\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:36.509966 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:36.509797 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l6qcv\" (UID: \"8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l6qcv" Oct 30 06:56:36.510023 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:36.509986 4951 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 06:56:36.510078 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:36.510055 4951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-cluster-monitoring-operator-tls podName:8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:38.510034729 +0000 UTC m=+4.853082678 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-l6qcv" (UID: "8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2") : secret "cluster-monitoring-operator-tls" not found Oct 30 06:56:36.510139 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:36.510106 4951 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 06:56:36.510139 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:36.510129 4951 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 06:56:36.510231 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:36.510140 4951 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b5f74d469-87gxs: secret "image-registry-tls" not found Oct 30 06:56:36.510231 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:36.510151 4951 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 06:56:36.510231 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:36.510169 4951 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dcb9bff4d-5xf9f: secret "image-registry-tls" not found Oct 30 06:56:36.510376 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:36.510153 4951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d930accb-e4d8-404b-8155-1b6a8edd812e-metrics-tls podName:d930accb-e4d8-404b-8155-1b6a8edd812e nodeName:}" failed. No retries permitted until 2026-10-30 06:56:38.510139804 +0000 UTC m=+4.853187768 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d930accb-e4d8-404b-8155-1b6a8edd812e-metrics-tls") pod "dns-default-vgrqz" (UID: "d930accb-e4d8-404b-8155-1b6a8edd812e") : secret "dns-default-metrics-tls" not found Oct 30 06:56:36.510376 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:36.510271 4951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls podName:b9d1faf4-274f-49c8-9340-f85463840b27 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:38.510256408 +0000 UTC m=+4.853304360 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls") pod "image-registry-5b5f74d469-87gxs" (UID: "b9d1faf4-274f-49c8-9340-f85463840b27") : secret "image-registry-tls" not found Oct 30 06:56:36.510376 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:36.510287 4951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-service-ca-bundle podName:0c49a4a4-033f-4a62-8bc4-6b59531e8bf7 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:38.510278002 +0000 UTC m=+4.853325947 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-service-ca-bundle") pod "router-default-7f6bb8cb6f-fzpwq" (UID: "0c49a4a4-033f-4a62-8bc4-6b59531e8bf7") : configmap references non-existent config key: service-ca.crt Oct 30 06:56:36.510376 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:36.510304 4951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls podName:36b00a60-14fb-4364-8943-6bcacc5365cc nodeName:}" failed. No retries permitted until 2026-10-30 06:56:38.510294058 +0000 UTC m=+4.853342000 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls") pod "image-registry-5dcb9bff4d-5xf9f" (UID: "36b00a60-14fb-4364-8943-6bcacc5365cc") : secret "image-registry-tls" not found Oct 30 06:56:36.512305 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:36.512283 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r5bpp\" (UID: \"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3\") " pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:36.512790 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:36.512764 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1b14112-6c97-4bfa-8f2a-5988f59ad06a-metrics-certs\") pod \"network-metrics-daemon-g6vbv\" (UID: \"f1b14112-6c97-4bfa-8f2a-5988f59ad06a\") " pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:56:36.597080 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:36.597004 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6vbv" Oct 30 06:56:36.602433 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:36.602407 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r5bpp" Oct 30 06:56:36.836886 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:36.836859 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/2.log" Oct 30 06:56:36.837365 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:36.837304 4951 scope.go:122] "RemoveContainer" containerID="6c0a7f3d03d409278e136b4be6b4fd9b9861fc108c08e90429cb106204060163" Oct 30 06:56:36.837584 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:36.837536 4951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zw4km_openshift-console-operator(42802760-36a8-4b79-8690-5a00528503c7)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" podUID="42802760-36a8-4b79-8690-5a00528503c7" Oct 30 06:56:36.839636 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:36.839607 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-cjn9t" event={"ID":"5943f8dc-3e86-49fa-ac40-70a1f11fb69e","Type":"ContainerStarted","Data":"8718949112d7a4804d703ab8daa567b33e1be02a5ceaa8d1d149d46ecae80a3a"} Oct 30 06:56:36.912614 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:36.912586 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g6vbv"] Oct 30 06:56:36.914166 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:36.914143 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b14112_6c97_4bfa_8f2a_5988f59ad06a.slice/crio-17f6894aeb662b771b252967ac820c989a7510b776370e7d02de41206e5246da WatchSource:0}: Error finding container 17f6894aeb662b771b252967ac820c989a7510b776370e7d02de41206e5246da: Status 404 returned error can't find the container with id 17f6894aeb662b771b252967ac820c989a7510b776370e7d02de41206e5246da Oct 30 06:56:36.927036 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:36.927013 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r5bpp"] Oct 30 06:56:36.928591 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:36.928554 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c8581ea_c199_4fcf_8f45_f9ef6c9cc2e3.slice/crio-dbbca26f3c342dd4fd677b9b8fc03569451d94730b4265cf19d69627161686e9 WatchSource:0}: Error finding container dbbca26f3c342dd4fd677b9b8fc03569451d94730b4265cf19d69627161686e9: Status 404 returned error can't find the container with id dbbca26f3c342dd4fd677b9b8fc03569451d94730b4265cf19d69627161686e9 Oct 30 06:56:37.849252 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:37.849216 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g6vbv" event={"ID":"f1b14112-6c97-4bfa-8f2a-5988f59ad06a","Type":"ContainerStarted","Data":"17f6894aeb662b771b252967ac820c989a7510b776370e7d02de41206e5246da"} Oct 30 06:56:37.851499 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:37.851473 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cflrf" event={"ID":"50ab5bd7-ea08-460b-9932-1dac54f3005b","Type":"ContainerStarted","Data":"b8c746d41de420dd3bb8a78ac6825cbac1bd533c13dd489541546c4b7031085e"} Oct 30 06:56:37.851629 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:37.851524 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cflrf" event={"ID":"50ab5bd7-ea08-460b-9932-1dac54f3005b","Type":"ContainerStarted","Data":"7a7ab2a5b3def6b31b0a29af19ac3b724b9a20ca0db74a48c81db349a6ccb216"} Oct 30 06:56:37.853730 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:37.853704 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r5bpp" event={"ID":"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3","Type":"ContainerStarted","Data":"cbfce8076cf6848fff5204460b63519f5e12387a2970cefd3dcdc25b1810f619"} Oct 30 06:56:37.853831 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:37.853773 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r5bpp" event={"ID":"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3","Type":"ContainerStarted","Data":"e896bc44e40bec2803b8ffcab39bad1c8138979d1112febd6f95e98c8b7554fb"} Oct 30 06:56:37.853831 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:37.853806 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r5bpp" event={"ID":"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3","Type":"ContainerStarted","Data":"dbbca26f3c342dd4fd677b9b8fc03569451d94730b4265cf19d69627161686e9"} Oct 30 06:56:38.516693 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:38.516660 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rjdmc" Oct 30 06:56:38.527976 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:38.527941 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:38.528155 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:38.527991 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d930accb-e4d8-404b-8155-1b6a8edd812e-metrics-tls\") pod \"dns-default-vgrqz\" (UID: \"d930accb-e4d8-404b-8155-1b6a8edd812e\") " pod="openshift-dns/dns-default-vgrqz" Oct 30 06:56:38.528155 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:38.528030 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:38.528155 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:38.528056 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-service-ca-bundle\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:38.528155 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:38.528089 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l6qcv\" (UID: \"8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l6qcv" Oct 30 06:56:38.528155 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:38.528107 4951 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 06:56:38.528155 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:38.528128 4951 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dcb9bff4d-5xf9f: secret "image-registry-tls" not found Oct 30 06:56:38.528416 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:38.528166 4951 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 06:56:38.528416 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:38.528185 4951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls podName:36b00a60-14fb-4364-8943-6bcacc5365cc nodeName:}" failed. No retries permitted until 2026-10-30 06:56:42.528164806 +0000 UTC m=+8.871212751 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls") pod "image-registry-5dcb9bff4d-5xf9f" (UID: "36b00a60-14fb-4364-8943-6bcacc5365cc") : secret "image-registry-tls" not found Oct 30 06:56:38.528416 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:38.528200 4951 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 06:56:38.528416 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:38.528204 4951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-service-ca-bundle podName:0c49a4a4-033f-4a62-8bc4-6b59531e8bf7 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:42.528195466 +0000 UTC m=+8.871243409 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-service-ca-bundle") pod "router-default-7f6bb8cb6f-fzpwq" (UID: "0c49a4a4-033f-4a62-8bc4-6b59531e8bf7") : configmap references non-existent config key: service-ca.crt Oct 30 06:56:38.528416 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:38.528207 4951 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 06:56:38.528416 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:38.528222 4951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d930accb-e4d8-404b-8155-1b6a8edd812e-metrics-tls podName:d930accb-e4d8-404b-8155-1b6a8edd812e nodeName:}" failed. No retries permitted until 2026-10-30 06:56:42.528214389 +0000 UTC m=+8.871262333 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d930accb-e4d8-404b-8155-1b6a8edd812e-metrics-tls") pod "dns-default-vgrqz" (UID: "d930accb-e4d8-404b-8155-1b6a8edd812e") : secret "dns-default-metrics-tls" not found Oct 30 06:56:38.528416 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:38.528224 4951 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b5f74d469-87gxs: secret "image-registry-tls" not found Oct 30 06:56:38.528416 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:38.528247 4951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-cluster-monitoring-operator-tls podName:8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:42.528232343 +0000 UTC m=+8.871280300 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-l6qcv" (UID: "8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2") : secret "cluster-monitoring-operator-tls" not found Oct 30 06:56:38.528416 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:38.528264 4951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls podName:b9d1faf4-274f-49c8-9340-f85463840b27 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:42.528254991 +0000 UTC m=+8.871302939 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls") pod "image-registry-5b5f74d469-87gxs" (UID: "b9d1faf4-274f-49c8-9340-f85463840b27") : secret "image-registry-tls" not found Oct 30 06:56:38.842040 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:38.841965 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qq6f5_4664d551-2cbf-42c5-a730-0d8c088f4f0f/dns-node-resolver/0.log" Oct 30 06:56:38.859776 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:38.859739 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g6vbv" event={"ID":"f1b14112-6c97-4bfa-8f2a-5988f59ad06a","Type":"ContainerStarted","Data":"bfda77cf8518bcd20fefc08c23f636e99841ec3c3a4fac2472086261e07f5032"} Oct 30 06:56:38.860267 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:38.859809 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g6vbv" event={"ID":"f1b14112-6c97-4bfa-8f2a-5988f59ad06a","Type":"ContainerStarted","Data":"20fda3edb3b0c02e0490258e5bdb063e88d38957f19a48add84177ad4a30c3b9"} Oct 30 06:56:39.842195 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:39.842167 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9fpvq_d1b84f62-7f09-42d0-8ed1-a2447efa3bf3/node-ca/0.log" Oct 30 06:56:39.865307 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:39.865267 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r5bpp" event={"ID":"4c8581ea-c199-4fcf-8f45-f9ef6c9cc2e3","Type":"ContainerStarted","Data":"77b568de37c03a8f17b1037a716ba02f398074949f0891af8ed231f0d139dc25"} Oct 30 06:56:40.151254 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:40.151153 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" Oct 30 06:56:40.151645 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:40.151626 4951 scope.go:122] "RemoveContainer" containerID="6c0a7f3d03d409278e136b4be6b4fd9b9861fc108c08e90429cb106204060163" Oct 30 06:56:40.151845 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:40.151825 4951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zw4km_openshift-console-operator(42802760-36a8-4b79-8690-5a00528503c7)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" podUID="42802760-36a8-4b79-8690-5a00528503c7" Oct 30 06:56:40.643122 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:40.643090 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qrksf_1ab9750b-0582-4c7c-b770-b1b7fbe8994c/serve-healthcheck-canary/0.log" Oct 30 06:56:41.043334 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:41.043315 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-6t2h6_1327b481-6353-4d2d-8847-caa7fc7e3a3d/migrator/0.log" Oct 30 06:56:41.310872 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:41.310797 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:41.311064 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:41.311049 4951 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 06:56:41.444175 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:41.444145 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-8xtng_20a9a026-75f0-4e90-b66a-066d3b80facf/kube-storage-version-migrator-operator/0.log" Oct 30 06:56:42.564256 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:42.564217 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:42.564660 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:42.564258 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-service-ca-bundle\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:42.564660 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:42.564297 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l6qcv\" (UID: \"8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l6qcv" Oct 30 06:56:42.564660 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:42.564383 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:42.564660 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:42.564416 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d930accb-e4d8-404b-8155-1b6a8edd812e-metrics-tls\") pod \"dns-default-vgrqz\" (UID: \"d930accb-e4d8-404b-8155-1b6a8edd812e\") " pod="openshift-dns/dns-default-vgrqz" Oct 30 06:56:42.565094 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:42.565063 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c49a4a4-033f-4a62-8bc4-6b59531e8bf7-service-ca-bundle\") pod \"router-default-7f6bb8cb6f-fzpwq\" (UID: \"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7\") " pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:42.566820 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:42.566793 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-l6qcv\" (UID: \"8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l6qcv" Oct 30 06:56:42.566941 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:42.566882 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d930accb-e4d8-404b-8155-1b6a8edd812e-metrics-tls\") pod \"dns-default-vgrqz\" (UID: \"d930accb-e4d8-404b-8155-1b6a8edd812e\") " pod="openshift-dns/dns-default-vgrqz" Oct 30 06:56:42.567011 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:42.566982 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls\") pod \"image-registry-5b5f74d469-87gxs\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:42.567073 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:42.566982 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls\") pod \"image-registry-5dcb9bff4d-5xf9f\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:42.587927 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:42.587893 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l6qcv" Oct 30 06:56:42.591323 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:42.591300 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:42.592380 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:42.592356 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:42.600392 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:42.600372 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vgrqz" Oct 30 06:56:42.600494 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:42.600469 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:42.826682 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:42.826608 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8tlbb" Oct 30 06:56:42.929097 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:42.929068 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7f6bb8cb6f-fzpwq"] Oct 30 06:56:42.930774 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:42.930675 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c49a4a4_033f_4a62_8bc4_6b59531e8bf7.slice/crio-f1da32e112c37931bbabafbf1f5914e1bd35d5587c3dd44e7cb3aff4a8ce06f5 WatchSource:0}: Error finding container f1da32e112c37931bbabafbf1f5914e1bd35d5587c3dd44e7cb3aff4a8ce06f5: Status 404 returned error can't find the container with id f1da32e112c37931bbabafbf1f5914e1bd35d5587c3dd44e7cb3aff4a8ce06f5 Oct 30 06:56:42.931752 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:42.931731 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b5f74d469-87gxs"] Oct 30 06:56:42.933475 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:42.933356 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d1faf4_274f_49c8_9340_f85463840b27.slice/crio-9f5e9fbb9c613c42fae6025c50e64e0eeae188234c9434b4a812dd9463f86f50 WatchSource:0}: Error finding container 9f5e9fbb9c613c42fae6025c50e64e0eeae188234c9434b4a812dd9463f86f50: Status 404 returned error can't find the container with id 9f5e9fbb9c613c42fae6025c50e64e0eeae188234c9434b4a812dd9463f86f50 Oct 30 06:56:42.935760 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:42.935738 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-l6qcv"] Oct 30 06:56:42.936946 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:42.936923 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ffc7ac6_2db9_4391_8fd5_5e78ba03a4c2.slice/crio-c0c95bd799d02e5f2f184131bc9f48413334926926a23975e3a79baff7278805 WatchSource:0}: Error finding container c0c95bd799d02e5f2f184131bc9f48413334926926a23975e3a79baff7278805: Status 404 returned error can't find the container with id c0c95bd799d02e5f2f184131bc9f48413334926926a23975e3a79baff7278805 Oct 30 06:56:42.939479 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:42.939427 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vgrqz"] Oct 30 06:56:42.941515 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:42.941494 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5dcb9bff4d-5xf9f"] Oct 30 06:56:42.946244 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:42.946211 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b00a60_14fb_4364_8943_6bcacc5365cc.slice/crio-d768984ed96c0ef3a125fdc834120e60fa62ae96aac95b85b4629f0ea49bb159 WatchSource:0}: Error finding container d768984ed96c0ef3a125fdc834120e60fa62ae96aac95b85b4629f0ea49bb159: Status 404 returned error can't find the container with id d768984ed96c0ef3a125fdc834120e60fa62ae96aac95b85b4629f0ea49bb159 Oct 30 06:56:43.881074 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:43.881034 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vgrqz" event={"ID":"d930accb-e4d8-404b-8155-1b6a8edd812e","Type":"ContainerStarted","Data":"4e2173780f4372b7ad0a2bf9602a7caccdd9eaf63fa6677a21478edf8eb47f30"} Oct 30 06:56:43.882892 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:43.882859 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" event={"ID":"36b00a60-14fb-4364-8943-6bcacc5365cc","Type":"ContainerStarted","Data":"dd071a8624630f1a8c9b13cd92a10ca8575dc157828ec8ac18e8a942db947105"} Oct 30 06:56:43.883028 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:43.882908 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" event={"ID":"36b00a60-14fb-4364-8943-6bcacc5365cc","Type":"ContainerStarted","Data":"d768984ed96c0ef3a125fdc834120e60fa62ae96aac95b85b4629f0ea49bb159"} Oct 30 06:56:43.885427 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:43.885400 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" event={"ID":"b9d1faf4-274f-49c8-9340-f85463840b27","Type":"ContainerStarted","Data":"addca0f18ba4624cff1968f82de056e3fc6a26d5c12bf2a19bca3fc4938fc427"} Oct 30 06:56:43.885526 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:43.885448 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" event={"ID":"b9d1faf4-274f-49c8-9340-f85463840b27","Type":"ContainerStarted","Data":"9f5e9fbb9c613c42fae6025c50e64e0eeae188234c9434b4a812dd9463f86f50"} Oct 30 06:56:43.886961 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:43.886938 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l6qcv" event={"ID":"8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2","Type":"ContainerStarted","Data":"c0c95bd799d02e5f2f184131bc9f48413334926926a23975e3a79baff7278805"} Oct 30 06:56:43.888756 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:43.888730 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" event={"ID":"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7","Type":"ContainerStarted","Data":"d2a9e29327e30c14e20b1d395640077207fe8370b9a9e177ca562e75da1abaa9"} Oct 30 06:56:43.888855 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:43.888771 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" event={"ID":"0c49a4a4-033f-4a62-8bc4-6b59531e8bf7","Type":"ContainerStarted","Data":"f1da32e112c37931bbabafbf1f5914e1bd35d5587c3dd44e7cb3aff4a8ce06f5"} Oct 30 06:56:44.304444 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:44.304405 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:56:44.591548 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:44.591495 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:44.594758 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:44.594723 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:44.894299 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:44.894145 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-l6qcv" event={"ID":"8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2","Type":"ContainerStarted","Data":"e08ced79de5031cd6d439d96a5bfff8c675b5f981620a2feb669b27e826a639d"} Oct 30 06:56:44.896468 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:44.896442 4951 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 06:56:44.896627 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:44.896473 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vgrqz" event={"ID":"d930accb-e4d8-404b-8155-1b6a8edd812e","Type":"ContainerStarted","Data":"f87af40d8b354c4bd333c7cbb45032f50a043bddc062ab7677f26f8a411fc317"} Oct 30 06:56:44.896627 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:44.896527 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vgrqz" event={"ID":"d930accb-e4d8-404b-8155-1b6a8edd812e","Type":"ContainerStarted","Data":"6f57ca649969c7b01d41c5fc636f3538b6dd77a2e02eba39ebc234c005bb1e83"} Oct 30 06:56:44.896627 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:44.896616 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vgrqz" Oct 30 06:56:45.900877 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:45.900850 4951 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 06:56:46.764505 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:46.764469 4951 scope.go:122] "RemoveContainer" containerID="6c0a7f3d03d409278e136b4be6b4fd9b9861fc108c08e90429cb106204060163" Oct 30 06:56:46.913103 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:46.913076 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/3.log" Oct 30 06:56:46.913503 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:46.913487 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/2.log" Oct 30 06:56:46.913540 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:46.913519 4951 generic.go:356] "Generic (PLEG): container finished" podID="42802760-36a8-4b79-8690-5a00528503c7" containerID="aef69f621627983c2bfb6b4a164899185ca5750a375202b13d65bc2153dee7ed" exitCode=255 Oct 30 06:56:46.913586 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:46.913560 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" event={"ID":"42802760-36a8-4b79-8690-5a00528503c7","Type":"ContainerDied","Data":"aef69f621627983c2bfb6b4a164899185ca5750a375202b13d65bc2153dee7ed"} Oct 30 06:56:46.913655 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:46.913644 4951 scope.go:122] "RemoveContainer" containerID="6c0a7f3d03d409278e136b4be6b4fd9b9861fc108c08e90429cb106204060163" Oct 30 06:56:46.913955 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:46.913937 4951 scope.go:122] "RemoveContainer" containerID="aef69f621627983c2bfb6b4a164899185ca5750a375202b13d65bc2153dee7ed" Oct 30 06:56:46.914147 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:46.914129 4951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zw4km_openshift-console-operator(42802760-36a8-4b79-8690-5a00528503c7)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" podUID="42802760-36a8-4b79-8690-5a00528503c7" Oct 30 06:56:47.159881 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:47.159810 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:56:47.919669 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:47.919640 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/3.log" Oct 30 06:56:50.097466 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:50.097429 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b5f74d469-87gxs"] Oct 30 06:56:50.152075 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:50.152043 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" Oct 30 06:56:50.152425 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:50.152407 4951 scope.go:122] "RemoveContainer" containerID="aef69f621627983c2bfb6b4a164899185ca5750a375202b13d65bc2153dee7ed" Oct 30 06:56:50.152655 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:50.152633 4951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zw4km_openshift-console-operator(42802760-36a8-4b79-8690-5a00528503c7)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" podUID="42802760-36a8-4b79-8690-5a00528503c7" Oct 30 06:56:50.279801 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:50.279775 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mjwrd" Oct 30 06:56:51.199410 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:51.199376 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:51.199863 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:51.199491 4951 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 06:56:51.200654 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:51.200628 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7f6bb8cb6f-fzpwq" Oct 30 06:56:53.023002 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:53.022963 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 06:56:54.903635 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:54.903606 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vgrqz" Oct 30 06:56:56.293926 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:56.293885 4951 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" Oct 30 06:56:56.294424 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:56.294353 4951 scope.go:122] "RemoveContainer" containerID="aef69f621627983c2bfb6b4a164899185ca5750a375202b13d65bc2153dee7ed" Oct 30 06:56:56.294637 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:56:56.294552 4951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zw4km_openshift-console-operator(42802760-36a8-4b79-8690-5a00528503c7)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" podUID="42802760-36a8-4b79-8690-5a00528503c7" Oct 30 06:56:57.164463 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.164424 4951 patch_prober.go:29] interesting pod/image-registry-5dcb9bff4d-5xf9f container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 06:56:57.164671 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.164517 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" podUID="36b00a60-14fb-4364-8943-6bcacc5365cc" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:56:57.484363 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.484277 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-q7nb2"] Oct 30 06:56:57.514103 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.514064 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q7nb2" Oct 30 06:56:57.546372 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.546335 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 06:56:57.546372 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.546335 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gbbvr\"" Oct 30 06:56:57.546372 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.546340 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 06:56:57.588089 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.588060 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 06:56:57.588281 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.588265 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 06:56:57.591542 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.591522 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f800ff9e-dc3a-4d35-a111-93f13e27e594-node-exporter-wtmp\") pod \"node-exporter-q7nb2\" (UID: \"f800ff9e-dc3a-4d35-a111-93f13e27e594\") " pod="openshift-monitoring/node-exporter-q7nb2" Oct 30 06:56:57.591606 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.591558 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f800ff9e-dc3a-4d35-a111-93f13e27e594-metrics-client-ca\") pod \"node-exporter-q7nb2\" (UID: \"f800ff9e-dc3a-4d35-a111-93f13e27e594\") " pod="openshift-monitoring/node-exporter-q7nb2" Oct 30 06:56:57.591691 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.591671 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f800ff9e-dc3a-4d35-a111-93f13e27e594-node-exporter-tls\") pod \"node-exporter-q7nb2\" (UID: \"f800ff9e-dc3a-4d35-a111-93f13e27e594\") " pod="openshift-monitoring/node-exporter-q7nb2" Oct 30 06:56:57.591764 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.591743 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f800ff9e-dc3a-4d35-a111-93f13e27e594-sys\") pod \"node-exporter-q7nb2\" (UID: \"f800ff9e-dc3a-4d35-a111-93f13e27e594\") " pod="openshift-monitoring/node-exporter-q7nb2" Oct 30 06:56:57.591796 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.591777 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f800ff9e-dc3a-4d35-a111-93f13e27e594-root\") pod \"node-exporter-q7nb2\" (UID: \"f800ff9e-dc3a-4d35-a111-93f13e27e594\") " pod="openshift-monitoring/node-exporter-q7nb2" Oct 30 06:56:57.591828 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.591799 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggkrp\" (UniqueName: \"kubernetes.io/projected/f800ff9e-dc3a-4d35-a111-93f13e27e594-kube-api-access-ggkrp\") pod \"node-exporter-q7nb2\" (UID: \"f800ff9e-dc3a-4d35-a111-93f13e27e594\") " pod="openshift-monitoring/node-exporter-q7nb2" Oct 30 06:56:57.591861 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.591824 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f800ff9e-dc3a-4d35-a111-93f13e27e594-node-exporter-textfile\") pod \"node-exporter-q7nb2\" (UID: \"f800ff9e-dc3a-4d35-a111-93f13e27e594\") " pod="openshift-monitoring/node-exporter-q7nb2" Oct 30 06:56:57.591901 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.591884 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f800ff9e-dc3a-4d35-a111-93f13e27e594-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q7nb2\" (UID: \"f800ff9e-dc3a-4d35-a111-93f13e27e594\") " pod="openshift-monitoring/node-exporter-q7nb2" Oct 30 06:56:57.591951 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.591939 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f800ff9e-dc3a-4d35-a111-93f13e27e594-node-exporter-accelerators-collector-config\") pod \"node-exporter-q7nb2\" (UID: \"f800ff9e-dc3a-4d35-a111-93f13e27e594\") " pod="openshift-monitoring/node-exporter-q7nb2" Oct 30 06:56:57.692992 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.692953 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f800ff9e-dc3a-4d35-a111-93f13e27e594-node-exporter-wtmp\") pod \"node-exporter-q7nb2\" (UID: \"f800ff9e-dc3a-4d35-a111-93f13e27e594\") " pod="openshift-monitoring/node-exporter-q7nb2" Oct 30 06:56:57.692992 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.692995 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f800ff9e-dc3a-4d35-a111-93f13e27e594-metrics-client-ca\") pod \"node-exporter-q7nb2\" (UID: \"f800ff9e-dc3a-4d35-a111-93f13e27e594\") " pod="openshift-monitoring/node-exporter-q7nb2" Oct 30 06:56:57.693212 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.693046 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f800ff9e-dc3a-4d35-a111-93f13e27e594-node-exporter-tls\") pod \"node-exporter-q7nb2\" (UID: \"f800ff9e-dc3a-4d35-a111-93f13e27e594\") " pod="openshift-monitoring/node-exporter-q7nb2" Oct 30 06:56:57.693212 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.693078 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f800ff9e-dc3a-4d35-a111-93f13e27e594-sys\") pod \"node-exporter-q7nb2\" (UID: \"f800ff9e-dc3a-4d35-a111-93f13e27e594\") " pod="openshift-monitoring/node-exporter-q7nb2" Oct 30 06:56:57.693212 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.693094 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f800ff9e-dc3a-4d35-a111-93f13e27e594-root\") pod \"node-exporter-q7nb2\" (UID: \"f800ff9e-dc3a-4d35-a111-93f13e27e594\") " pod="openshift-monitoring/node-exporter-q7nb2" Oct 30 06:56:57.693212 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.693148 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f800ff9e-dc3a-4d35-a111-93f13e27e594-root\") pod \"node-exporter-q7nb2\" (UID: \"f800ff9e-dc3a-4d35-a111-93f13e27e594\") " pod="openshift-monitoring/node-exporter-q7nb2" Oct 30 06:56:57.693212 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.693152 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f800ff9e-dc3a-4d35-a111-93f13e27e594-sys\") pod \"node-exporter-q7nb2\" (UID: \"f800ff9e-dc3a-4d35-a111-93f13e27e594\") " pod="openshift-monitoring/node-exporter-q7nb2" Oct 30 06:56:57.693212 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.693160 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggkrp\" (UniqueName: \"kubernetes.io/projected/f800ff9e-dc3a-4d35-a111-93f13e27e594-kube-api-access-ggkrp\") pod \"node-exporter-q7nb2\" (UID: \"f800ff9e-dc3a-4d35-a111-93f13e27e594\") " pod="openshift-monitoring/node-exporter-q7nb2" Oct 30 06:56:57.693212 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.693207 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f800ff9e-dc3a-4d35-a111-93f13e27e594-node-exporter-textfile\") pod \"node-exporter-q7nb2\" (UID: \"f800ff9e-dc3a-4d35-a111-93f13e27e594\") " pod="openshift-monitoring/node-exporter-q7nb2" Oct 30 06:56:57.693639 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.693161 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f800ff9e-dc3a-4d35-a111-93f13e27e594-node-exporter-wtmp\") pod \"node-exporter-q7nb2\" (UID: \"f800ff9e-dc3a-4d35-a111-93f13e27e594\") " pod="openshift-monitoring/node-exporter-q7nb2" Oct 30 06:56:57.693639 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.693239 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f800ff9e-dc3a-4d35-a111-93f13e27e594-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q7nb2\" (UID: \"f800ff9e-dc3a-4d35-a111-93f13e27e594\") " pod="openshift-monitoring/node-exporter-q7nb2" Oct 30 06:56:57.693639 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.693276 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f800ff9e-dc3a-4d35-a111-93f13e27e594-node-exporter-accelerators-collector-config\") pod \"node-exporter-q7nb2\" (UID: \"f800ff9e-dc3a-4d35-a111-93f13e27e594\") " pod="openshift-monitoring/node-exporter-q7nb2" Oct 30 06:56:57.693761 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.693634 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f800ff9e-dc3a-4d35-a111-93f13e27e594-node-exporter-textfile\") pod \"node-exporter-q7nb2\" (UID: \"f800ff9e-dc3a-4d35-a111-93f13e27e594\") " pod="openshift-monitoring/node-exporter-q7nb2" Oct 30 06:56:57.693761 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.693694 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f800ff9e-dc3a-4d35-a111-93f13e27e594-metrics-client-ca\") pod \"node-exporter-q7nb2\" (UID: \"f800ff9e-dc3a-4d35-a111-93f13e27e594\") " pod="openshift-monitoring/node-exporter-q7nb2" Oct 30 06:56:57.693856 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.693826 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f800ff9e-dc3a-4d35-a111-93f13e27e594-node-exporter-accelerators-collector-config\") pod \"node-exporter-q7nb2\" (UID: \"f800ff9e-dc3a-4d35-a111-93f13e27e594\") " pod="openshift-monitoring/node-exporter-q7nb2" Oct 30 06:56:57.695634 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.695614 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f800ff9e-dc3a-4d35-a111-93f13e27e594-node-exporter-tls\") pod \"node-exporter-q7nb2\" (UID: \"f800ff9e-dc3a-4d35-a111-93f13e27e594\") " pod="openshift-monitoring/node-exporter-q7nb2" Oct 30 06:56:57.695720 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.695702 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f800ff9e-dc3a-4d35-a111-93f13e27e594-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q7nb2\" (UID: \"f800ff9e-dc3a-4d35-a111-93f13e27e594\") " pod="openshift-monitoring/node-exporter-q7nb2" Oct 30 06:56:57.718073 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.718039 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggkrp\" (UniqueName: \"kubernetes.io/projected/f800ff9e-dc3a-4d35-a111-93f13e27e594-kube-api-access-ggkrp\") pod \"node-exporter-q7nb2\" (UID: \"f800ff9e-dc3a-4d35-a111-93f13e27e594\") " pod="openshift-monitoring/node-exporter-q7nb2" Oct 30 06:56:57.740766 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.740702 4951 patch_prober.go:29] interesting pod/image-registry-5dcb9bff4d-5xf9f container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 06:56:57.740886 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.740767 4951 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" podUID="36b00a60-14fb-4364-8943-6bcacc5365cc" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:56:57.875380 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.875342 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q7nb2" Oct 30 06:56:57.912538 ip-10-0-132-173 kubenswrapper[4951]: W1030 06:56:57.912509 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf800ff9e_dc3a_4d35_a111_93f13e27e594.slice/crio-c0d6d45964b12a814f1d66104afc1479c01041a1046e6509cd02fc83f3c4bcc5 WatchSource:0}: Error finding container c0d6d45964b12a814f1d66104afc1479c01041a1046e6509cd02fc83f3c4bcc5: Status 404 returned error can't find the container with id c0d6d45964b12a814f1d66104afc1479c01041a1046e6509cd02fc83f3c4bcc5 Oct 30 06:56:57.958647 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:57.958610 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q7nb2" event={"ID":"f800ff9e-dc3a-4d35-a111-93f13e27e594","Type":"ContainerStarted","Data":"c0d6d45964b12a814f1d66104afc1479c01041a1046e6509cd02fc83f3c4bcc5"} Oct 30 06:56:59.968081 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:59.968048 4951 generic.go:356] "Generic (PLEG): container finished" podID="f800ff9e-dc3a-4d35-a111-93f13e27e594" containerID="f1a5b036b1c25ed9658c390d0fbc29ae782d7bf0078a758cd57b3f3a9ea6ce24" exitCode=0 Oct 30 06:56:59.968443 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:56:59.968135 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q7nb2" event={"ID":"f800ff9e-dc3a-4d35-a111-93f13e27e594","Type":"ContainerDied","Data":"f1a5b036b1c25ed9658c390d0fbc29ae782d7bf0078a758cd57b3f3a9ea6ce24"} Oct 30 06:57:00.104997 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:00.104968 4951 patch_prober.go:29] interesting pod/image-registry-5b5f74d469-87gxs container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 06:57:00.105131 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:00.105022 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" podUID="b9d1faf4-274f-49c8-9340-f85463840b27" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:57:00.977564 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:00.977530 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q7nb2" event={"ID":"f800ff9e-dc3a-4d35-a111-93f13e27e594","Type":"ContainerStarted","Data":"d515a56cc80441527c7a0e84620c088129155e7238dbc5d5090d3b24cb08467a"} Oct 30 06:57:00.977956 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:00.977642 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q7nb2" event={"ID":"f800ff9e-dc3a-4d35-a111-93f13e27e594","Type":"ContainerStarted","Data":"efc08f9d45fd9409cab6749423a3ded41245bc754646dd980661660ce67f54b8"} Oct 30 06:57:01.049348 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:01.049295 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-q7nb2" podStartSLOduration=2.678497523 podStartE2EDuration="4.049283717s" totalImagesPullingTime="1.290840388s" totalInitContainerRuntime="79.945806ms" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:01.036719227 +0000 UTC m=+27.379767193" watchObservedRunningTime="2026-10-30 06:57:01.049283717 +0000 UTC m=+27.392331682" Oct 30 06:57:07.164343 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:07.164313 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:57:08.520304 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:08.520272 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rjdmc" Oct 30 06:57:09.764161 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:09.764122 4951 scope.go:122] "RemoveContainer" containerID="aef69f621627983c2bfb6b4a164899185ca5750a375202b13d65bc2153dee7ed" Oct 30 06:57:10.014288 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:10.014212 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/3.log" Oct 30 06:57:10.014441 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:10.014298 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" event={"ID":"42802760-36a8-4b79-8690-5a00528503c7","Type":"ContainerStarted","Data":"be83f5a5dcf11c3e8f5abf8faf999821945ff29f0eb0ed5e01404cbcb7b196c5"} Oct 30 06:57:10.014654 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:10.014636 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" Oct 30 06:57:10.101636 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:10.101608 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:57:10.134964 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:10.134933 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zw4km" Oct 30 06:57:12.164310 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:12.164275 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5dcb9bff4d-5xf9f"] Oct 30 06:57:15.118311 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:15.118257 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" podUID="b9d1faf4-274f-49c8-9340-f85463840b27" containerName="registry" containerID="cri-o://addca0f18ba4624cff1968f82de056e3fc6a26d5c12bf2a19bca3fc4938fc427" gracePeriod=30 Oct 30 06:57:15.388083 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:15.388008 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:57:15.435324 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:15.435281 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-bound-sa-token\") pod \"b9d1faf4-274f-49c8-9340-f85463840b27\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " Oct 30 06:57:15.435500 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:15.435376 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9d1faf4-274f-49c8-9340-f85463840b27-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9d1faf4-274f-49c8-9340-f85463840b27-installation-pull-secrets\") pod \"b9d1faf4-274f-49c8-9340-f85463840b27\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " Oct 30 06:57:15.435500 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:15.435400 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls\" (UniqueName: \"kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls\") pod \"b9d1faf4-274f-49c8-9340-f85463840b27\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " Oct 30 06:57:15.435500 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:15.435444 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9d1faf4-274f-49c8-9340-f85463840b27-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9d1faf4-274f-49c8-9340-f85463840b27-ca-trust-extracted\") pod \"b9d1faf4-274f-49c8-9340-f85463840b27\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " Oct 30 06:57:15.435500 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:15.435474 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9d1faf4-274f-49c8-9340-f85463840b27-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9d1faf4-274f-49c8-9340-f85463840b27-image-registry-private-configuration\") pod \"b9d1faf4-274f-49c8-9340-f85463840b27\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " Oct 30 06:57:15.435746 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:15.435508 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9d1faf4-274f-49c8-9340-f85463840b27-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9d1faf4-274f-49c8-9340-f85463840b27-trusted-ca\") pod \"b9d1faf4-274f-49c8-9340-f85463840b27\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " Oct 30 06:57:15.435746 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:15.435527 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-kube-api-access-rrc6h\" (UniqueName: \"kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-kube-api-access-rrc6h\") pod \"b9d1faf4-274f-49c8-9340-f85463840b27\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " Oct 30 06:57:15.435746 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:15.435551 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9d1faf4-274f-49c8-9340-f85463840b27-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9d1faf4-274f-49c8-9340-f85463840b27-registry-certificates\") pod \"b9d1faf4-274f-49c8-9340-f85463840b27\" (UID: \"b9d1faf4-274f-49c8-9340-f85463840b27\") " Oct 30 06:57:15.436412 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:15.436058 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9d1faf4-274f-49c8-9340-f85463840b27-trusted-ca" pod "b9d1faf4-274f-49c8-9340-f85463840b27" (UID: "b9d1faf4-274f-49c8-9340-f85463840b27"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:15.436412 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:15.436111 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9d1faf4-274f-49c8-9340-f85463840b27-registry-certificates" pod "b9d1faf4-274f-49c8-9340-f85463840b27" (UID: "b9d1faf4-274f-49c8-9340-f85463840b27"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:15.438260 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:15.438211 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls" pod "b9d1faf4-274f-49c8-9340-f85463840b27" (UID: "b9d1faf4-274f-49c8-9340-f85463840b27"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:15.438350 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:15.438258 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9d1faf4-274f-49c8-9340-f85463840b27-installation-pull-secrets" pod "b9d1faf4-274f-49c8-9340-f85463840b27" (UID: "b9d1faf4-274f-49c8-9340-f85463840b27"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:15.438393 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:15.438377 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-bound-sa-token" pod "b9d1faf4-274f-49c8-9340-f85463840b27" (UID: "b9d1faf4-274f-49c8-9340-f85463840b27"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:15.438610 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:15.438567 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-kube-api-access-rrc6h" pod "b9d1faf4-274f-49c8-9340-f85463840b27" (UID: "b9d1faf4-274f-49c8-9340-f85463840b27"). InnerVolumeSpecName "kube-api-access-rrc6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:15.438694 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:15.438662 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9d1faf4-274f-49c8-9340-f85463840b27-image-registry-private-configuration" pod "b9d1faf4-274f-49c8-9340-f85463840b27" (UID: "b9d1faf4-274f-49c8-9340-f85463840b27"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:15.442813 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:15.442793 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d1faf4-274f-49c8-9340-f85463840b27-ca-trust-extracted" pod "b9d1faf4-274f-49c8-9340-f85463840b27" (UID: "b9d1faf4-274f-49c8-9340-f85463840b27"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:15.536374 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:15.536335 4951 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-bound-sa-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 06:57:15.536374 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:15.536369 4951 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9d1faf4-274f-49c8-9340-f85463840b27-installation-pull-secrets\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 06:57:15.536374 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:15.536381 4951 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-registry-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 06:57:15.536662 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:15.536391 4951 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9d1faf4-274f-49c8-9340-f85463840b27-ca-trust-extracted\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 06:57:15.536662 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:15.536402 4951 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9d1faf4-274f-49c8-9340-f85463840b27-image-registry-private-configuration\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 06:57:15.536662 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:15.536411 4951 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9d1faf4-274f-49c8-9340-f85463840b27-trusted-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 06:57:15.536662 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:15.536420 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrc6h\" (UniqueName: \"kubernetes.io/projected/b9d1faf4-274f-49c8-9340-f85463840b27-kube-api-access-rrc6h\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 06:57:15.536662 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:15.536428 4951 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9d1faf4-274f-49c8-9340-f85463840b27-registry-certificates\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 06:57:16.038942 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:16.038855 4951 generic.go:356] "Generic (PLEG): container finished" podID="b9d1faf4-274f-49c8-9340-f85463840b27" containerID="addca0f18ba4624cff1968f82de056e3fc6a26d5c12bf2a19bca3fc4938fc427" exitCode=0 Oct 30 06:57:16.038942 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:16.038911 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" event={"ID":"b9d1faf4-274f-49c8-9340-f85463840b27","Type":"ContainerDied","Data":"addca0f18ba4624cff1968f82de056e3fc6a26d5c12bf2a19bca3fc4938fc427"} Oct 30 06:57:16.038942 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:16.038923 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" Oct 30 06:57:16.039197 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:16.038958 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b5f74d469-87gxs" event={"ID":"b9d1faf4-274f-49c8-9340-f85463840b27","Type":"ContainerDied","Data":"9f5e9fbb9c613c42fae6025c50e64e0eeae188234c9434b4a812dd9463f86f50"} Oct 30 06:57:16.039197 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:16.038984 4951 scope.go:122] "RemoveContainer" containerID="addca0f18ba4624cff1968f82de056e3fc6a26d5c12bf2a19bca3fc4938fc427" Oct 30 06:57:16.047082 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:16.047051 4951 scope.go:122] "RemoveContainer" containerID="addca0f18ba4624cff1968f82de056e3fc6a26d5c12bf2a19bca3fc4938fc427" Oct 30 06:57:16.047345 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:57:16.047323 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"addca0f18ba4624cff1968f82de056e3fc6a26d5c12bf2a19bca3fc4938fc427\": container with ID starting with addca0f18ba4624cff1968f82de056e3fc6a26d5c12bf2a19bca3fc4938fc427 not found: ID does not exist" containerID="addca0f18ba4624cff1968f82de056e3fc6a26d5c12bf2a19bca3fc4938fc427" Oct 30 06:57:16.047415 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:16.047358 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"addca0f18ba4624cff1968f82de056e3fc6a26d5c12bf2a19bca3fc4938fc427"} err="failed to get container status \"addca0f18ba4624cff1968f82de056e3fc6a26d5c12bf2a19bca3fc4938fc427\": rpc error: code = NotFound desc = could not find container \"addca0f18ba4624cff1968f82de056e3fc6a26d5c12bf2a19bca3fc4938fc427\": container with ID starting with addca0f18ba4624cff1968f82de056e3fc6a26d5c12bf2a19bca3fc4938fc427 not found: ID does not exist" Oct 30 06:57:16.102886 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:16.102862 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b5f74d469-87gxs"] Oct 30 06:57:16.115444 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:16.115421 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5b5f74d469-87gxs"] Oct 30 06:57:17.769719 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:17.769683 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9d1faf4-274f-49c8-9340-f85463840b27" path="/var/lib/kubelet/pods/b9d1faf4-274f-49c8-9340-f85463840b27/volumes" Oct 30 06:57:23.022825 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:57:23.022782 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 06:57:37.184531 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:37.184476 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" podUID="36b00a60-14fb-4364-8943-6bcacc5365cc" containerName="registry" containerID="cri-o://dd071a8624630f1a8c9b13cd92a10ca8575dc157828ec8ac18e8a942db947105" gracePeriod=30 Oct 30 06:57:37.428590 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:37.428542 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:57:37.506362 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:37.506333 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36b00a60-14fb-4364-8943-6bcacc5365cc-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/36b00a60-14fb-4364-8943-6bcacc5365cc-trusted-ca\") pod \"36b00a60-14fb-4364-8943-6bcacc5365cc\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " Oct 30 06:57:37.506533 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:37.506374 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36b00a60-14fb-4364-8943-6bcacc5365cc-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/36b00a60-14fb-4364-8943-6bcacc5365cc-installation-pull-secrets\") pod \"36b00a60-14fb-4364-8943-6bcacc5365cc\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " Oct 30 06:57:37.506533 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:37.506402 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-kube-api-access-s88rb\" (UniqueName: \"kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-kube-api-access-s88rb\") pod \"36b00a60-14fb-4364-8943-6bcacc5365cc\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " Oct 30 06:57:37.506533 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:37.506430 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b00a60-14fb-4364-8943-6bcacc5365cc-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/36b00a60-14fb-4364-8943-6bcacc5365cc-ca-trust-extracted\") pod \"36b00a60-14fb-4364-8943-6bcacc5365cc\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " Oct 30 06:57:37.506533 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:37.506460 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-bound-sa-token\") pod \"36b00a60-14fb-4364-8943-6bcacc5365cc\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " Oct 30 06:57:37.506533 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:37.506494 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-certificates\") pod \"36b00a60-14fb-4364-8943-6bcacc5365cc\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " Oct 30 06:57:37.506838 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:37.506582 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36b00a60-14fb-4364-8943-6bcacc5365cc-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/36b00a60-14fb-4364-8943-6bcacc5365cc-image-registry-private-configuration\") pod \"36b00a60-14fb-4364-8943-6bcacc5365cc\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " Oct 30 06:57:37.506838 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:37.506611 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls\" (UniqueName: \"kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls\") pod \"36b00a60-14fb-4364-8943-6bcacc5365cc\" (UID: \"36b00a60-14fb-4364-8943-6bcacc5365cc\") " Oct 30 06:57:37.507689 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:37.507655 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36b00a60-14fb-4364-8943-6bcacc5365cc-trusted-ca" pod "36b00a60-14fb-4364-8943-6bcacc5365cc" (UID: "36b00a60-14fb-4364-8943-6bcacc5365cc"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:37.507889 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:37.507858 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-certificates" pod "36b00a60-14fb-4364-8943-6bcacc5365cc" (UID: "36b00a60-14fb-4364-8943-6bcacc5365cc"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:37.509859 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:37.509818 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-kube-api-access-s88rb" pod "36b00a60-14fb-4364-8943-6bcacc5365cc" (UID: "36b00a60-14fb-4364-8943-6bcacc5365cc"). InnerVolumeSpecName "kube-api-access-s88rb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:37.509859 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:37.509837 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36b00a60-14fb-4364-8943-6bcacc5365cc-image-registry-private-configuration" pod "36b00a60-14fb-4364-8943-6bcacc5365cc" (UID: "36b00a60-14fb-4364-8943-6bcacc5365cc"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:37.509982 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:37.509855 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-bound-sa-token" pod "36b00a60-14fb-4364-8943-6bcacc5365cc" (UID: "36b00a60-14fb-4364-8943-6bcacc5365cc"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:37.509982 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:37.509873 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls" pod "36b00a60-14fb-4364-8943-6bcacc5365cc" (UID: "36b00a60-14fb-4364-8943-6bcacc5365cc"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:37.513683 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:37.513653 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b00a60-14fb-4364-8943-6bcacc5365cc-ca-trust-extracted" pod "36b00a60-14fb-4364-8943-6bcacc5365cc" (UID: "36b00a60-14fb-4364-8943-6bcacc5365cc"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:37.513683 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:37.513667 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36b00a60-14fb-4364-8943-6bcacc5365cc-installation-pull-secrets" pod "36b00a60-14fb-4364-8943-6bcacc5365cc" (UID: "36b00a60-14fb-4364-8943-6bcacc5365cc"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:37.607200 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:37.607166 4951 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/36b00a60-14fb-4364-8943-6bcacc5365cc-trusted-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 06:57:37.607200 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:37.607193 4951 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/36b00a60-14fb-4364-8943-6bcacc5365cc-installation-pull-secrets\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 06:57:37.607200 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:37.607206 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s88rb\" (UniqueName: \"kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-kube-api-access-s88rb\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 06:57:37.607433 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:37.607214 4951 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/36b00a60-14fb-4364-8943-6bcacc5365cc-ca-trust-extracted\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 06:57:37.607433 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:37.607223 4951 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-bound-sa-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 06:57:37.607433 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:37.607231 4951 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-certificates\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 06:57:37.607433 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:37.607242 4951 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/36b00a60-14fb-4364-8943-6bcacc5365cc-image-registry-private-configuration\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 06:57:37.607433 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:37.607252 4951 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36b00a60-14fb-4364-8943-6bcacc5365cc-registry-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 06:57:38.120049 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:38.119964 4951 generic.go:356] "Generic (PLEG): container finished" podID="36b00a60-14fb-4364-8943-6bcacc5365cc" containerID="dd071a8624630f1a8c9b13cd92a10ca8575dc157828ec8ac18e8a942db947105" exitCode=0 Oct 30 06:57:38.120049 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:38.120010 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" Oct 30 06:57:38.120283 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:38.120051 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" event={"ID":"36b00a60-14fb-4364-8943-6bcacc5365cc","Type":"ContainerDied","Data":"dd071a8624630f1a8c9b13cd92a10ca8575dc157828ec8ac18e8a942db947105"} Oct 30 06:57:38.120283 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:38.120110 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dcb9bff4d-5xf9f" event={"ID":"36b00a60-14fb-4364-8943-6bcacc5365cc","Type":"ContainerDied","Data":"d768984ed96c0ef3a125fdc834120e60fa62ae96aac95b85b4629f0ea49bb159"} Oct 30 06:57:38.120283 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:38.120139 4951 scope.go:122] "RemoveContainer" containerID="dd071a8624630f1a8c9b13cd92a10ca8575dc157828ec8ac18e8a942db947105" Oct 30 06:57:38.128333 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:38.128314 4951 scope.go:122] "RemoveContainer" containerID="dd071a8624630f1a8c9b13cd92a10ca8575dc157828ec8ac18e8a942db947105" Oct 30 06:57:38.128638 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:57:38.128613 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd071a8624630f1a8c9b13cd92a10ca8575dc157828ec8ac18e8a942db947105\": container with ID starting with dd071a8624630f1a8c9b13cd92a10ca8575dc157828ec8ac18e8a942db947105 not found: ID does not exist" containerID="dd071a8624630f1a8c9b13cd92a10ca8575dc157828ec8ac18e8a942db947105" Oct 30 06:57:38.128733 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:38.128645 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd071a8624630f1a8c9b13cd92a10ca8575dc157828ec8ac18e8a942db947105"} err="failed to get container status \"dd071a8624630f1a8c9b13cd92a10ca8575dc157828ec8ac18e8a942db947105\": rpc error: code = NotFound desc = could not find container \"dd071a8624630f1a8c9b13cd92a10ca8575dc157828ec8ac18e8a942db947105\": container with ID starting with dd071a8624630f1a8c9b13cd92a10ca8575dc157828ec8ac18e8a942db947105 not found: ID does not exist" Oct 30 06:57:38.184780 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:38.184745 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5dcb9bff4d-5xf9f"] Oct 30 06:57:38.197364 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:38.197328 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5dcb9bff4d-5xf9f"] Oct 30 06:57:39.768069 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:39.768038 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36b00a60-14fb-4364-8943-6bcacc5365cc" path="/var/lib/kubelet/pods/36b00a60-14fb-4364-8943-6bcacc5365cc/volumes" Oct 30 06:57:50.163741 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:50.163704 4951 generic.go:356] "Generic (PLEG): container finished" podID="4b59e668-d5a4-4ce0-a1e9-a16d63f51990" containerID="71efa7e0684157232e61eaaf212a9c38bb649bb585571d2d04f9bbb2e3cec4d3" exitCode=0 Oct 30 06:57:50.164116 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:50.163775 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-qqt8g" event={"ID":"4b59e668-d5a4-4ce0-a1e9-a16d63f51990","Type":"ContainerDied","Data":"71efa7e0684157232e61eaaf212a9c38bb649bb585571d2d04f9bbb2e3cec4d3"} Oct 30 06:57:50.164161 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:50.164137 4951 scope.go:122] "RemoveContainer" containerID="71efa7e0684157232e61eaaf212a9c38bb649bb585571d2d04f9bbb2e3cec4d3" Oct 30 06:57:51.169326 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:57:51.169290 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-qqt8g" event={"ID":"4b59e668-d5a4-4ce0-a1e9-a16d63f51990","Type":"ContainerStarted","Data":"9b9a9129a3b9d0cf06288d8babba2807ea347add8db62561f72cdf12cc330b10"} Oct 30 06:57:53.020427 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:57:53.020389 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 06:58:00.200700 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:58:00.200667 4951 generic.go:356] "Generic (PLEG): container finished" podID="20a9a026-75f0-4e90-b66a-066d3b80facf" containerID="561f1d8ed87b1ca5db56adba97783fd27d143734c74e9bb0263e95f1f45620bd" exitCode=0 Oct 30 06:58:00.201115 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:58:00.200738 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8xtng" event={"ID":"20a9a026-75f0-4e90-b66a-066d3b80facf","Type":"ContainerDied","Data":"561f1d8ed87b1ca5db56adba97783fd27d143734c74e9bb0263e95f1f45620bd"} Oct 30 06:58:00.201115 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:58:00.201049 4951 scope.go:122] "RemoveContainer" containerID="561f1d8ed87b1ca5db56adba97783fd27d143734c74e9bb0263e95f1f45620bd" Oct 30 06:58:01.205911 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:58:01.205877 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-8xtng" event={"ID":"20a9a026-75f0-4e90-b66a-066d3b80facf","Type":"ContainerStarted","Data":"db18b4032b614cda024da8fbadbe0812aefcbaf9005df8690135a60af0199cfe"} Oct 30 06:58:04.218022 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:58:04.217992 4951 generic.go:356] "Generic (PLEG): container finished" podID="c5f1f83f-9bfd-45f5-b065-ba95cf659956" containerID="81f3c61a54268340de3e4cc760b6ba988c88d5bc62a23656f86998d0205394cd" exitCode=0 Oct 30 06:58:04.218492 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:58:04.218074 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2kt77" event={"ID":"c5f1f83f-9bfd-45f5-b065-ba95cf659956","Type":"ContainerDied","Data":"81f3c61a54268340de3e4cc760b6ba988c88d5bc62a23656f86998d0205394cd"} Oct 30 06:58:04.218492 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:58:04.218486 4951 scope.go:122] "RemoveContainer" containerID="81f3c61a54268340de3e4cc760b6ba988c88d5bc62a23656f86998d0205394cd" Oct 30 06:58:05.225838 ip-10-0-132-173 kubenswrapper[4951]: I1030 06:58:05.225801 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2kt77" event={"ID":"c5f1f83f-9bfd-45f5-b065-ba95cf659956","Type":"ContainerStarted","Data":"f7a94dbd707f646d28e01034d233b4909c0bb1b8904da58bd3d392bfd15ac053"} Oct 30 06:58:23.020938 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:58:23.020896 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 06:58:53.021422 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:58:53.021377 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 06:59:23.021820 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:59:23.021771 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 06:59:53.021771 ip-10-0-132-173 kubenswrapper[4951]: E1030 06:59:53.021733 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:00:23.021539 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:00:23.021460 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:00:53.023715 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:00:53.023676 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:01:23.020986 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:01:23.020945 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:01:33.778940 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:33.778910 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/3.log" Oct 30 07:01:33.780933 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:33.780911 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/3.log" Oct 30 07:01:33.783920 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:33.783901 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 07:01:33.785831 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:33.785813 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 07:01:34.593732 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.593701 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624"] Oct 30 07:01:34.594017 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.594002 4951 cpu_manager.go:414] "containerMap: removing container" podUID="b9d1faf4-274f-49c8-9340-f85463840b27" containerName="registry" Oct 30 07:01:34.594071 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.594019 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9d1faf4-274f-49c8-9340-f85463840b27" containerName="registry" Oct 30 07:01:34.594071 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.594030 4951 cpu_manager.go:414] "containerMap: removing container" podUID="36b00a60-14fb-4364-8943-6bcacc5365cc" containerName="registry" Oct 30 07:01:34.594071 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.594035 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36b00a60-14fb-4364-8943-6bcacc5365cc" containerName="registry" Oct 30 07:01:34.594161 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.594098 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9d1faf4-274f-49c8-9340-f85463840b27" containerName="registry" Oct 30 07:01:34.594161 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.594106 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="36b00a60-14fb-4364-8943-6bcacc5365cc" containerName="registry" Oct 30 07:01:34.597092 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.597063 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" Oct 30 07:01:34.613086 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.613054 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 07:01:34.613204 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.613187 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 07:01:34.613400 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.613387 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 07:01:34.613434 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.613417 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 07:01:34.623921 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.623902 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-725wz\"" Oct 30 07:01:34.625646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.625624 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624"] Oct 30 07:01:34.720192 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.720150 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-b9624\" (UID: \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" Oct 30 07:01:34.720374 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.720199 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-b9624\" (UID: \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" Oct 30 07:01:34.720374 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.720271 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jtmc\" (UniqueName: \"kubernetes.io/projected/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-kube-api-access-2jtmc\") pod \"istiod-openshift-gateway-7876bfcb74-b9624\" (UID: \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" Oct 30 07:01:34.720374 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.720343 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-b9624\" (UID: \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" Oct 30 07:01:34.720481 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.720390 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-b9624\" (UID: \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" Oct 30 07:01:34.720481 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.720411 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-b9624\" (UID: \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" Oct 30 07:01:34.720481 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.720427 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-b9624\" (UID: \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" Oct 30 07:01:34.821498 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.821468 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-b9624\" (UID: \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" Oct 30 07:01:34.821498 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.821500 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jtmc\" (UniqueName: \"kubernetes.io/projected/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-kube-api-access-2jtmc\") pod \"istiod-openshift-gateway-7876bfcb74-b9624\" (UID: \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" Oct 30 07:01:34.821994 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.821523 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-b9624\" (UID: \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" Oct 30 07:01:34.821994 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.821554 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-b9624\" (UID: \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" Oct 30 07:01:34.821994 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.821597 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-b9624\" (UID: \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" Oct 30 07:01:34.821994 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.821622 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-b9624\" (UID: \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" Oct 30 07:01:34.821994 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.821702 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-b9624\" (UID: \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" Oct 30 07:01:34.822235 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.822203 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-b9624\" (UID: \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" Oct 30 07:01:34.824067 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.824041 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-b9624\" (UID: \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" Oct 30 07:01:34.824155 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.824041 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-b9624\" (UID: \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" Oct 30 07:01:34.824289 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.824266 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-b9624\" (UID: \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" Oct 30 07:01:34.824392 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.824377 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-b9624\" (UID: \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" Oct 30 07:01:34.843384 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.843353 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-b9624\" (UID: \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" Oct 30 07:01:34.843651 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.843635 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jtmc\" (UniqueName: \"kubernetes.io/projected/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-kube-api-access-2jtmc\") pod \"istiod-openshift-gateway-7876bfcb74-b9624\" (UID: \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" Oct 30 07:01:34.908332 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:34.908273 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" Oct 30 07:01:35.211678 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:35.211594 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624"] Oct 30 07:01:35.215170 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:01:35.215142 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ef0885_9682_42f8_b4ff_f5b1bac6cbb4.slice/crio-7274741f7b88d170e783211a23716a5cd98d965725e09da7528fcadd2124c49d WatchSource:0}: Error finding container 7274741f7b88d170e783211a23716a5cd98d965725e09da7528fcadd2124c49d: Status 404 returned error can't find the container with id 7274741f7b88d170e783211a23716a5cd98d965725e09da7528fcadd2124c49d Oct 30 07:01:35.971684 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:35.971645 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" event={"ID":"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4","Type":"ContainerStarted","Data":"7274741f7b88d170e783211a23716a5cd98d965725e09da7528fcadd2124c49d"} Oct 30 07:01:37.661730 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:37.661019 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:01:37.661730 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:37.661105 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:01:37.981892 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:37.981811 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" event={"ID":"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4","Type":"ContainerStarted","Data":"26bba38b8dbb3df2a02aad0e098064673251694520708af203fe4d39770a24ba"} Oct 30 07:01:37.982076 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:37.982056 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" Oct 30 07:01:38.036351 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:38.036292 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" podStartSLOduration=1.593339532 podStartE2EDuration="4.036277358s" totalImagesPullingTime="2.442937826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:01:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:01:38.035532189 +0000 UTC m=+304.378580180" watchObservedRunningTime="2026-10-30 07:01:38.036277358 +0000 UTC m=+304.379325322" Oct 30 07:01:38.988136 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:38.988105 4951 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-b9624 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:01:38.988519 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:38.988157 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" podUID="95ef0885-9682-42f8-b4ff-f5b1bac6cbb4" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:01:41.987162 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:01:41.987132 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" Oct 30 07:01:53.022058 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:01:53.022022 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:02:21.006773 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:21.006736 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-scj2c"] Oct 30 07:02:21.010136 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:21.010109 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-scj2c" Oct 30 07:02:21.039071 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:21.039037 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-scj2c"] Oct 30 07:02:21.078596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:21.078537 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c01db306-a2a6-4fcc-98ee-d63bb8c3344a-tmp\") pod \"opendatahub-operator-controller-manager-77996896cc-scj2c\" (UID: \"c01db306-a2a6-4fcc-98ee-d63bb8c3344a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-scj2c" Oct 30 07:02:21.078805 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:21.078673 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c01db306-a2a6-4fcc-98ee-d63bb8c3344a-metrics-cert\") pod \"opendatahub-operator-controller-manager-77996896cc-scj2c\" (UID: \"c01db306-a2a6-4fcc-98ee-d63bb8c3344a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-scj2c" Oct 30 07:02:21.078805 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:21.078717 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c01db306-a2a6-4fcc-98ee-d63bb8c3344a-manifests\") pod \"opendatahub-operator-controller-manager-77996896cc-scj2c\" (UID: \"c01db306-a2a6-4fcc-98ee-d63bb8c3344a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-scj2c" Oct 30 07:02:21.078805 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:21.078745 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c01db306-a2a6-4fcc-98ee-d63bb8c3344a-webhook-cert\") pod \"opendatahub-operator-controller-manager-77996896cc-scj2c\" (UID: \"c01db306-a2a6-4fcc-98ee-d63bb8c3344a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-scj2c" Oct 30 07:02:21.078931 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:21.078851 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtrm8\" (UniqueName: \"kubernetes.io/projected/c01db306-a2a6-4fcc-98ee-d63bb8c3344a-kube-api-access-jtrm8\") pod \"opendatahub-operator-controller-manager-77996896cc-scj2c\" (UID: \"c01db306-a2a6-4fcc-98ee-d63bb8c3344a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-scj2c" Oct 30 07:02:21.078931 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:21.078899 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c01db306-a2a6-4fcc-98ee-d63bb8c3344a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77996896cc-scj2c\" (UID: \"c01db306-a2a6-4fcc-98ee-d63bb8c3344a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-scj2c" Oct 30 07:02:21.097995 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:21.097961 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 07:02:21.097995 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:21.097972 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 07:02:21.098181 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:21.097998 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 07:02:21.098181 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:21.097998 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 07:02:21.098181 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:21.098042 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-62jmt\"" Oct 30 07:02:21.098181 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:21.098073 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 07:02:21.179949 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:21.179913 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c01db306-a2a6-4fcc-98ee-d63bb8c3344a-metrics-cert\") pod \"opendatahub-operator-controller-manager-77996896cc-scj2c\" (UID: \"c01db306-a2a6-4fcc-98ee-d63bb8c3344a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-scj2c" Oct 30 07:02:21.179949 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:21.179952 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c01db306-a2a6-4fcc-98ee-d63bb8c3344a-manifests\") pod \"opendatahub-operator-controller-manager-77996896cc-scj2c\" (UID: \"c01db306-a2a6-4fcc-98ee-d63bb8c3344a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-scj2c" Oct 30 07:02:21.180234 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:21.179972 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c01db306-a2a6-4fcc-98ee-d63bb8c3344a-webhook-cert\") pod \"opendatahub-operator-controller-manager-77996896cc-scj2c\" (UID: \"c01db306-a2a6-4fcc-98ee-d63bb8c3344a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-scj2c" Oct 30 07:02:21.180234 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:21.180029 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtrm8\" (UniqueName: \"kubernetes.io/projected/c01db306-a2a6-4fcc-98ee-d63bb8c3344a-kube-api-access-jtrm8\") pod \"opendatahub-operator-controller-manager-77996896cc-scj2c\" (UID: \"c01db306-a2a6-4fcc-98ee-d63bb8c3344a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-scj2c" Oct 30 07:02:21.180234 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:21.180097 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c01db306-a2a6-4fcc-98ee-d63bb8c3344a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77996896cc-scj2c\" (UID: \"c01db306-a2a6-4fcc-98ee-d63bb8c3344a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-scj2c" Oct 30 07:02:21.180234 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:21.180147 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c01db306-a2a6-4fcc-98ee-d63bb8c3344a-tmp\") pod \"opendatahub-operator-controller-manager-77996896cc-scj2c\" (UID: \"c01db306-a2a6-4fcc-98ee-d63bb8c3344a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-scj2c" Oct 30 07:02:21.180462 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:21.180414 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c01db306-a2a6-4fcc-98ee-d63bb8c3344a-manifests\") pod \"opendatahub-operator-controller-manager-77996896cc-scj2c\" (UID: \"c01db306-a2a6-4fcc-98ee-d63bb8c3344a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-scj2c" Oct 30 07:02:21.182347 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:21.182326 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c01db306-a2a6-4fcc-98ee-d63bb8c3344a-tmp\") pod \"opendatahub-operator-controller-manager-77996896cc-scj2c\" (UID: \"c01db306-a2a6-4fcc-98ee-d63bb8c3344a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-scj2c" Oct 30 07:02:21.182636 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:21.182619 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c01db306-a2a6-4fcc-98ee-d63bb8c3344a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77996896cc-scj2c\" (UID: \"c01db306-a2a6-4fcc-98ee-d63bb8c3344a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-scj2c" Oct 30 07:02:21.182770 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:21.182750 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c01db306-a2a6-4fcc-98ee-d63bb8c3344a-metrics-cert\") pod \"opendatahub-operator-controller-manager-77996896cc-scj2c\" (UID: \"c01db306-a2a6-4fcc-98ee-d63bb8c3344a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-scj2c" Oct 30 07:02:21.182807 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:21.182750 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c01db306-a2a6-4fcc-98ee-d63bb8c3344a-webhook-cert\") pod \"opendatahub-operator-controller-manager-77996896cc-scj2c\" (UID: \"c01db306-a2a6-4fcc-98ee-d63bb8c3344a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-scj2c" Oct 30 07:02:21.203614 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:21.203586 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtrm8\" (UniqueName: \"kubernetes.io/projected/c01db306-a2a6-4fcc-98ee-d63bb8c3344a-kube-api-access-jtrm8\") pod \"opendatahub-operator-controller-manager-77996896cc-scj2c\" (UID: \"c01db306-a2a6-4fcc-98ee-d63bb8c3344a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-scj2c" Oct 30 07:02:21.330533 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:21.330451 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-scj2c" Oct 30 07:02:21.691812 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:21.691784 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-scj2c"] Oct 30 07:02:21.694351 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:02:21.694317 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc01db306_a2a6_4fcc_98ee_d63bb8c3344a.slice/crio-e37697b363ecdfe6aeaf37d6bd2e71d2c0ab19780a7763d9aef3267151e83003 WatchSource:0}: Error finding container e37697b363ecdfe6aeaf37d6bd2e71d2c0ab19780a7763d9aef3267151e83003: Status 404 returned error can't find the container with id e37697b363ecdfe6aeaf37d6bd2e71d2c0ab19780a7763d9aef3267151e83003 Oct 30 07:02:21.696121 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:21.696103 4951 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 07:02:22.138998 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:22.138959 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-scj2c" event={"ID":"c01db306-a2a6-4fcc-98ee-d63bb8c3344a","Type":"ContainerStarted","Data":"e37697b363ecdfe6aeaf37d6bd2e71d2c0ab19780a7763d9aef3267151e83003"} Oct 30 07:02:23.031512 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:02:23.031467 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:02:25.153704 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:25.153664 4951 generic.go:356] "Generic (PLEG): container finished" podID="c01db306-a2a6-4fcc-98ee-d63bb8c3344a" containerID="a32eb8d01a029ef46a7f9a6030f4f1a0dd904e23f8fb35cbe949d990f7886a76" exitCode=0 Oct 30 07:02:25.154136 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:25.153759 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-scj2c" event={"ID":"c01db306-a2a6-4fcc-98ee-d63bb8c3344a","Type":"ContainerDied","Data":"a32eb8d01a029ef46a7f9a6030f4f1a0dd904e23f8fb35cbe949d990f7886a76"} Oct 30 07:02:26.159847 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:26.159808 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-scj2c" event={"ID":"c01db306-a2a6-4fcc-98ee-d63bb8c3344a","Type":"ContainerStarted","Data":"abce6d4630aa39a264ea805ccf3f0cd8700d303a6c13b8e5bc69c8c24f0c6899"} Oct 30 07:02:26.160232 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:26.159906 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-scj2c" Oct 30 07:02:26.259108 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:26.259059 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-scj2c" podStartSLOduration=2.618171385 podStartE2EDuration="6.259046977s" totalImagesPullingTime="3.19834618s" totalInitContainerRuntime="442.529412ms" isStatefulPod=false podCreationTimestamp="2026-10-30 07:02:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:02:26.258846424 +0000 UTC m=+352.601894388" watchObservedRunningTime="2026-10-30 07:02:26.259046977 +0000 UTC m=+352.602094941" Oct 30 07:02:37.165993 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:37.165960 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-scj2c" Oct 30 07:02:53.022757 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:02:53.022718 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:02:56.340207 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:56.340169 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5cf9d6756f-sxt26"] Oct 30 07:02:56.343722 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:56.343702 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cf9d6756f-sxt26" Oct 30 07:02:56.404161 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:56.404130 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 07:02:56.404320 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:56.404208 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 07:02:56.404320 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:56.404276 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vptpj\"" Oct 30 07:02:56.405267 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:56.405245 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cf9d6756f-sxt26"] Oct 30 07:02:56.484442 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:56.484413 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c0b8673-ee48-45f2-8ea6-ac112d33cd51-tmp\") pod \"kube-auth-proxy-5cf9d6756f-sxt26\" (UID: \"9c0b8673-ee48-45f2-8ea6-ac112d33cd51\") " pod="openshift-ingress/kube-auth-proxy-5cf9d6756f-sxt26" Oct 30 07:02:56.484626 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:56.484455 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c0b8673-ee48-45f2-8ea6-ac112d33cd51-tls-certs\") pod \"kube-auth-proxy-5cf9d6756f-sxt26\" (UID: \"9c0b8673-ee48-45f2-8ea6-ac112d33cd51\") " pod="openshift-ingress/kube-auth-proxy-5cf9d6756f-sxt26" Oct 30 07:02:56.484684 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:56.484626 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29pkf\" (UniqueName: \"kubernetes.io/projected/9c0b8673-ee48-45f2-8ea6-ac112d33cd51-kube-api-access-29pkf\") pod \"kube-auth-proxy-5cf9d6756f-sxt26\" (UID: \"9c0b8673-ee48-45f2-8ea6-ac112d33cd51\") " pod="openshift-ingress/kube-auth-proxy-5cf9d6756f-sxt26" Oct 30 07:02:56.585832 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:56.585801 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29pkf\" (UniqueName: \"kubernetes.io/projected/9c0b8673-ee48-45f2-8ea6-ac112d33cd51-kube-api-access-29pkf\") pod \"kube-auth-proxy-5cf9d6756f-sxt26\" (UID: \"9c0b8673-ee48-45f2-8ea6-ac112d33cd51\") " pod="openshift-ingress/kube-auth-proxy-5cf9d6756f-sxt26" Oct 30 07:02:56.586022 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:56.585842 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c0b8673-ee48-45f2-8ea6-ac112d33cd51-tmp\") pod \"kube-auth-proxy-5cf9d6756f-sxt26\" (UID: \"9c0b8673-ee48-45f2-8ea6-ac112d33cd51\") " pod="openshift-ingress/kube-auth-proxy-5cf9d6756f-sxt26" Oct 30 07:02:56.586022 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:56.585872 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c0b8673-ee48-45f2-8ea6-ac112d33cd51-tls-certs\") pod \"kube-auth-proxy-5cf9d6756f-sxt26\" (UID: \"9c0b8673-ee48-45f2-8ea6-ac112d33cd51\") " pod="openshift-ingress/kube-auth-proxy-5cf9d6756f-sxt26" Oct 30 07:02:56.589031 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:56.589006 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c0b8673-ee48-45f2-8ea6-ac112d33cd51-tls-certs\") pod \"kube-auth-proxy-5cf9d6756f-sxt26\" (UID: \"9c0b8673-ee48-45f2-8ea6-ac112d33cd51\") " pod="openshift-ingress/kube-auth-proxy-5cf9d6756f-sxt26" Oct 30 07:02:56.590466 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:56.590420 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c0b8673-ee48-45f2-8ea6-ac112d33cd51-tmp\") pod \"kube-auth-proxy-5cf9d6756f-sxt26\" (UID: \"9c0b8673-ee48-45f2-8ea6-ac112d33cd51\") " pod="openshift-ingress/kube-auth-proxy-5cf9d6756f-sxt26" Oct 30 07:02:56.613601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:56.613552 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29pkf\" (UniqueName: \"kubernetes.io/projected/9c0b8673-ee48-45f2-8ea6-ac112d33cd51-kube-api-access-29pkf\") pod \"kube-auth-proxy-5cf9d6756f-sxt26\" (UID: \"9c0b8673-ee48-45f2-8ea6-ac112d33cd51\") " pod="openshift-ingress/kube-auth-proxy-5cf9d6756f-sxt26" Oct 30 07:02:56.655231 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:56.655200 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cf9d6756f-sxt26" Oct 30 07:02:56.950505 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:56.950471 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cf9d6756f-sxt26"] Oct 30 07:02:56.953318 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:02:56.953280 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c0b8673_ee48_45f2_8ea6_ac112d33cd51.slice/crio-b3ffd9070f0c1b2bbe58363bc479d290697dd502c56ef0560b6782764db94427 WatchSource:0}: Error finding container b3ffd9070f0c1b2bbe58363bc479d290697dd502c56ef0560b6782764db94427: Status 404 returned error can't find the container with id b3ffd9070f0c1b2bbe58363bc479d290697dd502c56ef0560b6782764db94427 Oct 30 07:02:57.282748 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:02:57.282709 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cf9d6756f-sxt26" event={"ID":"9c0b8673-ee48-45f2-8ea6-ac112d33cd51","Type":"ContainerStarted","Data":"b3ffd9070f0c1b2bbe58363bc479d290697dd502c56ef0560b6782764db94427"} Oct 30 07:03:00.302402 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:03:00.302358 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cf9d6756f-sxt26" event={"ID":"9c0b8673-ee48-45f2-8ea6-ac112d33cd51","Type":"ContainerStarted","Data":"d0cabeff34765ed5c9929665e1a89532d9f47c9c851b5d942739ce45b2b80281"} Oct 30 07:03:00.386814 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:03:00.386759 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5cf9d6756f-sxt26" podStartSLOduration=2.000420521 podStartE2EDuration="4.38674364s" totalImagesPullingTime="2.386323119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:02:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:03:00.385127552 +0000 UTC m=+386.728175514" watchObservedRunningTime="2026-10-30 07:03:00.38674364 +0000 UTC m=+386.729791605" Oct 30 07:03:00.411805 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:03:00.411767 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nk2g9"] Oct 30 07:03:00.415211 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:03:00.415192 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nk2g9" Oct 30 07:03:00.470056 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:03:00.470029 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 07:03:00.470425 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:03:00.470408 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 07:03:00.470667 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:03:00.470491 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tszrb\"" Oct 30 07:03:00.495877 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:03:00.495850 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nk2g9"] Oct 30 07:03:00.521706 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:03:00.521681 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhd8h\" (UniqueName: \"kubernetes.io/projected/454b78a6-5a90-41a9-8ff5-fa16edf2b257-kube-api-access-zhd8h\") pod \"kuadrant-operator-controller-manager-678d87766f-nk2g9\" (UID: \"454b78a6-5a90-41a9-8ff5-fa16edf2b257\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nk2g9" Oct 30 07:03:00.521817 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:03:00.521726 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/454b78a6-5a90-41a9-8ff5-fa16edf2b257-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-nk2g9\" (UID: \"454b78a6-5a90-41a9-8ff5-fa16edf2b257\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nk2g9" Oct 30 07:03:00.622936 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:03:00.622852 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/454b78a6-5a90-41a9-8ff5-fa16edf2b257-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-nk2g9\" (UID: \"454b78a6-5a90-41a9-8ff5-fa16edf2b257\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nk2g9" Oct 30 07:03:00.622936 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:03:00.622924 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhd8h\" (UniqueName: \"kubernetes.io/projected/454b78a6-5a90-41a9-8ff5-fa16edf2b257-kube-api-access-zhd8h\") pod \"kuadrant-operator-controller-manager-678d87766f-nk2g9\" (UID: \"454b78a6-5a90-41a9-8ff5-fa16edf2b257\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nk2g9" Oct 30 07:03:00.623283 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:03:00.623263 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/454b78a6-5a90-41a9-8ff5-fa16edf2b257-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-nk2g9\" (UID: \"454b78a6-5a90-41a9-8ff5-fa16edf2b257\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nk2g9" Oct 30 07:03:00.653054 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:03:00.653027 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhd8h\" (UniqueName: \"kubernetes.io/projected/454b78a6-5a90-41a9-8ff5-fa16edf2b257-kube-api-access-zhd8h\") pod \"kuadrant-operator-controller-manager-678d87766f-nk2g9\" (UID: \"454b78a6-5a90-41a9-8ff5-fa16edf2b257\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nk2g9" Oct 30 07:03:00.727272 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:03:00.727231 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nk2g9" Oct 30 07:03:01.049958 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:03:01.049923 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nk2g9"] Oct 30 07:03:01.063011 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:03:01.062983 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod454b78a6_5a90_41a9_8ff5_fa16edf2b257.slice/crio-157432484f6e71ad6192bf7a891e2637626b0e8c0b2cf71b6f3cb792b6628ff6 WatchSource:0}: Error finding container 157432484f6e71ad6192bf7a891e2637626b0e8c0b2cf71b6f3cb792b6628ff6: Status 404 returned error can't find the container with id 157432484f6e71ad6192bf7a891e2637626b0e8c0b2cf71b6f3cb792b6628ff6 Oct 30 07:03:01.307760 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:03:01.307675 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nk2g9" event={"ID":"454b78a6-5a90-41a9-8ff5-fa16edf2b257","Type":"ContainerStarted","Data":"157432484f6e71ad6192bf7a891e2637626b0e8c0b2cf71b6f3cb792b6628ff6"} Oct 30 07:03:05.329640 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:03:05.329602 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nk2g9" event={"ID":"454b78a6-5a90-41a9-8ff5-fa16edf2b257","Type":"ContainerStarted","Data":"57943a8311a17555ac8f9be5186f614ede1dbbb8751d0bdc93f40553b4d1f226"} Oct 30 07:03:05.330043 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:03:05.329799 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nk2g9" Oct 30 07:03:05.397928 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:03:05.397819 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nk2g9" podStartSLOduration=1.379336606 podStartE2EDuration="5.397806084s" totalImagesPullingTime="4.018469478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:03:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:03:05.386659551 +0000 UTC m=+391.729707516" watchObservedRunningTime="2026-10-30 07:03:05.397806084 +0000 UTC m=+391.740854048" Oct 30 07:03:16.336174 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:03:16.336094 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nk2g9" Oct 30 07:03:23.022543 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:03:23.022509 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:03:53.022205 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:03:53.022163 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:04:03.951231 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:03.951192 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz"] Oct 30 07:04:03.954008 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:03.953980 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz" Oct 30 07:04:03.959599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:03.959560 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d729858d-5129-43e9-b6fe-129b8267791e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ctldz\" (UID: \"d729858d-5129-43e9-b6fe-129b8267791e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz" Oct 30 07:04:03.959834 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:03.959807 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d729858d-5129-43e9-b6fe-129b8267791e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ctldz\" (UID: \"d729858d-5129-43e9-b6fe-129b8267791e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz" Oct 30 07:04:03.959945 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:03.959887 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99x2r\" (UniqueName: \"kubernetes.io/projected/d729858d-5129-43e9-b6fe-129b8267791e-kube-api-access-99x2r\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ctldz\" (UID: \"d729858d-5129-43e9-b6fe-129b8267791e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz" Oct 30 07:04:03.959945 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:03.959908 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d729858d-5129-43e9-b6fe-129b8267791e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ctldz\" (UID: \"d729858d-5129-43e9-b6fe-129b8267791e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz" Oct 30 07:04:03.960090 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:03.959998 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d729858d-5129-43e9-b6fe-129b8267791e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ctldz\" (UID: \"d729858d-5129-43e9-b6fe-129b8267791e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz" Oct 30 07:04:03.960090 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:03.960039 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d729858d-5129-43e9-b6fe-129b8267791e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ctldz\" (UID: \"d729858d-5129-43e9-b6fe-129b8267791e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz" Oct 30 07:04:03.960189 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:03.960089 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d729858d-5129-43e9-b6fe-129b8267791e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ctldz\" (UID: \"d729858d-5129-43e9-b6fe-129b8267791e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz" Oct 30 07:04:03.982923 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:03.982861 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz"] Oct 30 07:04:04.060910 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:04.060875 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d729858d-5129-43e9-b6fe-129b8267791e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ctldz\" (UID: \"d729858d-5129-43e9-b6fe-129b8267791e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz" Oct 30 07:04:04.061084 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:04.060917 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d729858d-5129-43e9-b6fe-129b8267791e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ctldz\" (UID: \"d729858d-5129-43e9-b6fe-129b8267791e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz" Oct 30 07:04:04.061084 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:04.060946 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d729858d-5129-43e9-b6fe-129b8267791e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ctldz\" (UID: \"d729858d-5129-43e9-b6fe-129b8267791e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz" Oct 30 07:04:04.061084 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:04.060961 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d729858d-5129-43e9-b6fe-129b8267791e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ctldz\" (UID: \"d729858d-5129-43e9-b6fe-129b8267791e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz" Oct 30 07:04:04.061084 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:04.060981 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d729858d-5129-43e9-b6fe-129b8267791e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ctldz\" (UID: \"d729858d-5129-43e9-b6fe-129b8267791e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz" Oct 30 07:04:04.061084 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:04.060999 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d729858d-5129-43e9-b6fe-129b8267791e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ctldz\" (UID: \"d729858d-5129-43e9-b6fe-129b8267791e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz" Oct 30 07:04:04.061084 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:04.061028 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99x2r\" (UniqueName: \"kubernetes.io/projected/d729858d-5129-43e9-b6fe-129b8267791e-kube-api-access-99x2r\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ctldz\" (UID: \"d729858d-5129-43e9-b6fe-129b8267791e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz" Oct 30 07:04:04.061821 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:04.061795 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d729858d-5129-43e9-b6fe-129b8267791e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ctldz\" (UID: \"d729858d-5129-43e9-b6fe-129b8267791e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz" Oct 30 07:04:04.063387 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:04.063363 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d729858d-5129-43e9-b6fe-129b8267791e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ctldz\" (UID: \"d729858d-5129-43e9-b6fe-129b8267791e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz" Oct 30 07:04:04.063491 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:04.063469 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d729858d-5129-43e9-b6fe-129b8267791e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ctldz\" (UID: \"d729858d-5129-43e9-b6fe-129b8267791e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz" Oct 30 07:04:04.064190 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:04.064169 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d729858d-5129-43e9-b6fe-129b8267791e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ctldz\" (UID: \"d729858d-5129-43e9-b6fe-129b8267791e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz" Oct 30 07:04:04.064241 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:04.064179 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d729858d-5129-43e9-b6fe-129b8267791e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ctldz\" (UID: \"d729858d-5129-43e9-b6fe-129b8267791e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz" Oct 30 07:04:04.082518 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:04.082499 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d729858d-5129-43e9-b6fe-129b8267791e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ctldz\" (UID: \"d729858d-5129-43e9-b6fe-129b8267791e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz" Oct 30 07:04:04.094315 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:04.094296 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99x2r\" (UniqueName: \"kubernetes.io/projected/d729858d-5129-43e9-b6fe-129b8267791e-kube-api-access-99x2r\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ctldz\" (UID: \"d729858d-5129-43e9-b6fe-129b8267791e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz" Oct 30 07:04:04.265620 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:04.265581 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz" Oct 30 07:04:04.570235 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:04.570149 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz"] Oct 30 07:04:04.573181 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:04:04.573156 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd729858d_5129_43e9_b6fe_129b8267791e.slice/crio-39d12021cc7e1038f824a2f2fa7a53561ab6e6e0ae8a554f24fe70773bda2bb0 WatchSource:0}: Error finding container 39d12021cc7e1038f824a2f2fa7a53561ab6e6e0ae8a554f24fe70773bda2bb0: Status 404 returned error can't find the container with id 39d12021cc7e1038f824a2f2fa7a53561ab6e6e0ae8a554f24fe70773bda2bb0 Oct 30 07:04:04.575157 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:04.575120 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:04:04.575216 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:04.575193 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:04:05.577188 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:05.577150 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz" event={"ID":"d729858d-5129-43e9-b6fe-129b8267791e","Type":"ContainerStarted","Data":"c8a3f9c150e1e5615497e53697ab440f07852e30a824f9676714da6fecf2b28e"} Oct 30 07:04:05.577693 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:05.577240 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz" event={"ID":"d729858d-5129-43e9-b6fe-129b8267791e","Type":"ContainerStarted","Data":"39d12021cc7e1038f824a2f2fa7a53561ab6e6e0ae8a554f24fe70773bda2bb0"} Oct 30 07:04:05.577693 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:05.577376 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz" Oct 30 07:04:05.578728 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:05.578709 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz" Oct 30 07:04:05.640989 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:05.640950 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz" podStartSLOduration=2.640938243 podStartE2EDuration="2.640938243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:04:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:04:05.629882676 +0000 UTC m=+451.972930652" watchObservedRunningTime="2026-10-30 07:04:05.640938243 +0000 UTC m=+451.983986208" Oct 30 07:04:05.705603 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:05.705557 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624"] Oct 30 07:04:05.705830 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:05.705791 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" podUID="95ef0885-9682-42f8-b4ff-f5b1bac6cbb4" containerName="discovery" containerID="cri-o://26bba38b8dbb3df2a02aad0e098064673251694520708af203fe4d39770a24ba" gracePeriod=30 Oct 30 07:04:05.950356 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:05.950329 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" Oct 30 07:04:05.974214 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:05.974180 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-cacerts\" (UniqueName: \"kubernetes.io/secret/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-cacerts\") pod \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\" (UID: \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\") " Oct 30 07:04:05.974405 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:05.974222 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-istio-kubeconfig\") pod \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\" (UID: \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\") " Oct 30 07:04:05.974405 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:05.974254 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-local-certs\") pod \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\" (UID: \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\") " Oct 30 07:04:05.974405 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:05.974277 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-istio-csr-ca-configmap\") pod \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\" (UID: \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\") " Oct 30 07:04:05.974405 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:05.974299 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-kube-api-access-2jtmc\" (UniqueName: \"kubernetes.io/projected/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-kube-api-access-2jtmc\") pod \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\" (UID: \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\") " Oct 30 07:04:05.974405 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:05.974323 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-istio-token\" (UniqueName: \"kubernetes.io/projected/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-istio-token\") pod \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\" (UID: \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\") " Oct 30 07:04:05.974683 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:05.974424 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-istio-csr-dns-cert\") pod \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\" (UID: \"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4\") " Oct 30 07:04:05.974760 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:05.974734 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-istio-csr-ca-configmap" pod "95ef0885-9682-42f8-b4ff-f5b1bac6cbb4" (UID: "95ef0885-9682-42f8-b4ff-f5b1bac6cbb4"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:04:05.976975 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:05.976945 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-local-certs" pod "95ef0885-9682-42f8-b4ff-f5b1bac6cbb4" (UID: "95ef0885-9682-42f8-b4ff-f5b1bac6cbb4"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:04:05.977102 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:05.976948 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-istio-kubeconfig" pod "95ef0885-9682-42f8-b4ff-f5b1bac6cbb4" (UID: "95ef0885-9682-42f8-b4ff-f5b1bac6cbb4"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:04:05.977187 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:05.977129 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-istio-csr-dns-cert" pod "95ef0885-9682-42f8-b4ff-f5b1bac6cbb4" (UID: "95ef0885-9682-42f8-b4ff-f5b1bac6cbb4"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:04:05.977187 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:05.977169 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-istio-token" pod "95ef0885-9682-42f8-b4ff-f5b1bac6cbb4" (UID: "95ef0885-9682-42f8-b4ff-f5b1bac6cbb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:04:05.977267 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:05.977186 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-kube-api-access-2jtmc" pod "95ef0885-9682-42f8-b4ff-f5b1bac6cbb4" (UID: "95ef0885-9682-42f8-b4ff-f5b1bac6cbb4"). InnerVolumeSpecName "kube-api-access-2jtmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:04:05.977267 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:05.977252 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-cacerts" pod "95ef0885-9682-42f8-b4ff-f5b1bac6cbb4" (UID: "95ef0885-9682-42f8-b4ff-f5b1bac6cbb4"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:04:06.075275 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:06.075226 4951 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-istio-csr-dns-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:04:06.075275 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:06.075261 4951 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-cacerts\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:04:06.075275 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:06.075277 4951 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-istio-kubeconfig\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:04:06.075275 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:06.075291 4951 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-local-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:04:06.075275 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:06.075303 4951 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-istio-csr-ca-configmap\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:04:06.075732 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:06.075314 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jtmc\" (UniqueName: \"kubernetes.io/projected/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-kube-api-access-2jtmc\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:04:06.075732 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:06.075327 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:04:06.582257 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:06.582219 4951 generic.go:356] "Generic (PLEG): container finished" podID="95ef0885-9682-42f8-b4ff-f5b1bac6cbb4" containerID="26bba38b8dbb3df2a02aad0e098064673251694520708af203fe4d39770a24ba" exitCode=0 Oct 30 07:04:06.582698 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:06.582272 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" Oct 30 07:04:06.582698 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:06.582292 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" event={"ID":"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4","Type":"ContainerDied","Data":"26bba38b8dbb3df2a02aad0e098064673251694520708af203fe4d39770a24ba"} Oct 30 07:04:06.582698 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:06.582348 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624" event={"ID":"95ef0885-9682-42f8-b4ff-f5b1bac6cbb4","Type":"ContainerDied","Data":"7274741f7b88d170e783211a23716a5cd98d965725e09da7528fcadd2124c49d"} Oct 30 07:04:06.582698 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:06.582370 4951 scope.go:122] "RemoveContainer" containerID="26bba38b8dbb3df2a02aad0e098064673251694520708af203fe4d39770a24ba" Oct 30 07:04:06.591978 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:06.591952 4951 scope.go:122] "RemoveContainer" containerID="26bba38b8dbb3df2a02aad0e098064673251694520708af203fe4d39770a24ba" Oct 30 07:04:06.592313 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:04:06.592289 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26bba38b8dbb3df2a02aad0e098064673251694520708af203fe4d39770a24ba\": container with ID starting with 26bba38b8dbb3df2a02aad0e098064673251694520708af203fe4d39770a24ba not found: ID does not exist" containerID="26bba38b8dbb3df2a02aad0e098064673251694520708af203fe4d39770a24ba" Oct 30 07:04:06.592365 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:06.592324 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26bba38b8dbb3df2a02aad0e098064673251694520708af203fe4d39770a24ba"} err="failed to get container status \"26bba38b8dbb3df2a02aad0e098064673251694520708af203fe4d39770a24ba\": rpc error: code = NotFound desc = could not find container \"26bba38b8dbb3df2a02aad0e098064673251694520708af203fe4d39770a24ba\": container with ID starting with 26bba38b8dbb3df2a02aad0e098064673251694520708af203fe4d39770a24ba not found: ID does not exist" Oct 30 07:04:06.629166 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:06.629131 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624"] Oct 30 07:04:06.640079 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:06.640049 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-b9624"] Oct 30 07:04:07.769593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:07.769533 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95ef0885-9682-42f8-b4ff-f5b1bac6cbb4" path="/var/lib/kubelet/pods/95ef0885-9682-42f8-b4ff-f5b1bac6cbb4/volumes" Oct 30 07:04:14.568178 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:14.568130 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8b74c94db-jqz4t"] Oct 30 07:04:14.568785 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:14.568580 4951 cpu_manager.go:414] "containerMap: removing container" podUID="95ef0885-9682-42f8-b4ff-f5b1bac6cbb4" containerName="discovery" Oct 30 07:04:14.568785 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:14.568595 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95ef0885-9682-42f8-b4ff-f5b1bac6cbb4" containerName="discovery" Oct 30 07:04:14.568785 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:14.568672 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="95ef0885-9682-42f8-b4ff-f5b1bac6cbb4" containerName="discovery" Oct 30 07:04:14.571443 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:14.571426 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8b74c94db-jqz4t" Oct 30 07:04:14.596225 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:14.596192 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:04:14.596411 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:14.596254 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 07:04:14.596411 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:14.596263 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 07:04:14.596411 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:14.596338 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:04:14.596411 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:14.596378 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-g865r\"" Oct 30 07:04:14.598231 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:14.598207 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8b74c94db-jqz4t"] Oct 30 07:04:14.645483 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:14.645444 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b040c64-4330-4e22-a88b-fa7ab9eac634-openshift-service-ca\") pod \"maas-api-8b74c94db-jqz4t\" (UID: \"6b040c64-4330-4e22-a88b-fa7ab9eac634\") " pod="odh-ai-gateway-infra/maas-api-8b74c94db-jqz4t" Oct 30 07:04:14.645483 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:14.645485 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b040c64-4330-4e22-a88b-fa7ab9eac634-maas-api-tls\") pod \"maas-api-8b74c94db-jqz4t\" (UID: \"6b040c64-4330-4e22-a88b-fa7ab9eac634\") " pod="odh-ai-gateway-infra/maas-api-8b74c94db-jqz4t" Oct 30 07:04:14.645719 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:14.645551 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b040c64-4330-4e22-a88b-fa7ab9eac634-metrics-certs\") pod \"maas-api-8b74c94db-jqz4t\" (UID: \"6b040c64-4330-4e22-a88b-fa7ab9eac634\") " pod="odh-ai-gateway-infra/maas-api-8b74c94db-jqz4t" Oct 30 07:04:14.645719 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:14.645616 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgg9j\" (UniqueName: \"kubernetes.io/projected/6b040c64-4330-4e22-a88b-fa7ab9eac634-kube-api-access-tgg9j\") pod \"maas-api-8b74c94db-jqz4t\" (UID: \"6b040c64-4330-4e22-a88b-fa7ab9eac634\") " pod="odh-ai-gateway-infra/maas-api-8b74c94db-jqz4t" Oct 30 07:04:14.746885 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:14.746844 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b040c64-4330-4e22-a88b-fa7ab9eac634-openshift-service-ca\") pod \"maas-api-8b74c94db-jqz4t\" (UID: \"6b040c64-4330-4e22-a88b-fa7ab9eac634\") " pod="odh-ai-gateway-infra/maas-api-8b74c94db-jqz4t" Oct 30 07:04:14.746885 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:14.746889 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b040c64-4330-4e22-a88b-fa7ab9eac634-maas-api-tls\") pod \"maas-api-8b74c94db-jqz4t\" (UID: \"6b040c64-4330-4e22-a88b-fa7ab9eac634\") " pod="odh-ai-gateway-infra/maas-api-8b74c94db-jqz4t" Oct 30 07:04:14.747103 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:14.746912 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b040c64-4330-4e22-a88b-fa7ab9eac634-metrics-certs\") pod \"maas-api-8b74c94db-jqz4t\" (UID: \"6b040c64-4330-4e22-a88b-fa7ab9eac634\") " pod="odh-ai-gateway-infra/maas-api-8b74c94db-jqz4t" Oct 30 07:04:14.747103 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:14.747084 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgg9j\" (UniqueName: \"kubernetes.io/projected/6b040c64-4330-4e22-a88b-fa7ab9eac634-kube-api-access-tgg9j\") pod \"maas-api-8b74c94db-jqz4t\" (UID: \"6b040c64-4330-4e22-a88b-fa7ab9eac634\") " pod="odh-ai-gateway-infra/maas-api-8b74c94db-jqz4t" Oct 30 07:04:14.747553 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:14.747529 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b040c64-4330-4e22-a88b-fa7ab9eac634-openshift-service-ca\") pod \"maas-api-8b74c94db-jqz4t\" (UID: \"6b040c64-4330-4e22-a88b-fa7ab9eac634\") " pod="odh-ai-gateway-infra/maas-api-8b74c94db-jqz4t" Oct 30 07:04:14.749383 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:14.749362 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b040c64-4330-4e22-a88b-fa7ab9eac634-metrics-certs\") pod \"maas-api-8b74c94db-jqz4t\" (UID: \"6b040c64-4330-4e22-a88b-fa7ab9eac634\") " pod="odh-ai-gateway-infra/maas-api-8b74c94db-jqz4t" Oct 30 07:04:14.749468 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:14.749451 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b040c64-4330-4e22-a88b-fa7ab9eac634-maas-api-tls\") pod \"maas-api-8b74c94db-jqz4t\" (UID: \"6b040c64-4330-4e22-a88b-fa7ab9eac634\") " pod="odh-ai-gateway-infra/maas-api-8b74c94db-jqz4t" Oct 30 07:04:14.769533 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:14.769500 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgg9j\" (UniqueName: \"kubernetes.io/projected/6b040c64-4330-4e22-a88b-fa7ab9eac634-kube-api-access-tgg9j\") pod \"maas-api-8b74c94db-jqz4t\" (UID: \"6b040c64-4330-4e22-a88b-fa7ab9eac634\") " pod="odh-ai-gateway-infra/maas-api-8b74c94db-jqz4t" Oct 30 07:04:14.884243 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:14.884148 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8b74c94db-jqz4t" Oct 30 07:04:15.177278 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:15.177197 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8b74c94db-jqz4t"] Oct 30 07:04:15.180416 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:04:15.180388 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b040c64_4330_4e22_a88b_fa7ab9eac634.slice/crio-ecc59e72d71d8aa8039f30d56adbf0337b4d29b715be2bd7fb0cec5196ba0647 WatchSource:0}: Error finding container ecc59e72d71d8aa8039f30d56adbf0337b4d29b715be2bd7fb0cec5196ba0647: Status 404 returned error can't find the container with id ecc59e72d71d8aa8039f30d56adbf0337b4d29b715be2bd7fb0cec5196ba0647 Oct 30 07:04:15.620901 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:15.620861 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8b74c94db-jqz4t" event={"ID":"6b040c64-4330-4e22-a88b-fa7ab9eac634","Type":"ContainerStarted","Data":"ecc59e72d71d8aa8039f30d56adbf0337b4d29b715be2bd7fb0cec5196ba0647"} Oct 30 07:04:17.630496 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:17.630456 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8b74c94db-jqz4t" event={"ID":"6b040c64-4330-4e22-a88b-fa7ab9eac634","Type":"ContainerStarted","Data":"9bd70a55f199e3169946b2a2b361a161bd01db00a46b7365e393d45275b8a8a5"} Oct 30 07:04:17.630936 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:17.630628 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8b74c94db-jqz4t" Oct 30 07:04:17.670966 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:17.670906 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8b74c94db-jqz4t" podStartSLOduration=1.400464242 podStartE2EDuration="3.670892289s" totalImagesPullingTime="2.270428047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:04:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:04:17.670164325 +0000 UTC m=+464.013212291" watchObservedRunningTime="2026-10-30 07:04:17.670892289 +0000 UTC m=+464.013940260" Oct 30 07:04:23.023199 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:04:23.023161 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:04:23.640488 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:23.640460 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8b74c94db-jqz4t" Oct 30 07:04:24.856394 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:24.856351 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-f785bd78b-t42sc"] Oct 30 07:04:24.859877 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:24.859860 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f785bd78b-t42sc" Oct 30 07:04:24.885892 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:24.885867 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f785bd78b-t42sc"] Oct 30 07:04:24.940042 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:24.940009 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eea3e53-4722-493d-8820-27d28c7e5270-openshift-service-ca\") pod \"maas-api-f785bd78b-t42sc\" (UID: \"9eea3e53-4722-493d-8820-27d28c7e5270\") " pod="odh-ai-gateway-infra/maas-api-f785bd78b-t42sc" Oct 30 07:04:24.940042 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:24.940043 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9eea3e53-4722-493d-8820-27d28c7e5270-maas-api-tls\") pod \"maas-api-f785bd78b-t42sc\" (UID: \"9eea3e53-4722-493d-8820-27d28c7e5270\") " pod="odh-ai-gateway-infra/maas-api-f785bd78b-t42sc" Oct 30 07:04:24.940236 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:24.940127 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l47wn\" (UniqueName: \"kubernetes.io/projected/9eea3e53-4722-493d-8820-27d28c7e5270-kube-api-access-l47wn\") pod \"maas-api-f785bd78b-t42sc\" (UID: \"9eea3e53-4722-493d-8820-27d28c7e5270\") " pod="odh-ai-gateway-infra/maas-api-f785bd78b-t42sc" Oct 30 07:04:24.940236 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:24.940175 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eea3e53-4722-493d-8820-27d28c7e5270-metrics-certs\") pod \"maas-api-f785bd78b-t42sc\" (UID: \"9eea3e53-4722-493d-8820-27d28c7e5270\") " pod="odh-ai-gateway-infra/maas-api-f785bd78b-t42sc" Oct 30 07:04:25.041275 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:25.041241 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eea3e53-4722-493d-8820-27d28c7e5270-openshift-service-ca\") pod \"maas-api-f785bd78b-t42sc\" (UID: \"9eea3e53-4722-493d-8820-27d28c7e5270\") " pod="odh-ai-gateway-infra/maas-api-f785bd78b-t42sc" Oct 30 07:04:25.041454 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:25.041285 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9eea3e53-4722-493d-8820-27d28c7e5270-maas-api-tls\") pod \"maas-api-f785bd78b-t42sc\" (UID: \"9eea3e53-4722-493d-8820-27d28c7e5270\") " pod="odh-ai-gateway-infra/maas-api-f785bd78b-t42sc" Oct 30 07:04:25.041519 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:25.041478 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l47wn\" (UniqueName: \"kubernetes.io/projected/9eea3e53-4722-493d-8820-27d28c7e5270-kube-api-access-l47wn\") pod \"maas-api-f785bd78b-t42sc\" (UID: \"9eea3e53-4722-493d-8820-27d28c7e5270\") " pod="odh-ai-gateway-infra/maas-api-f785bd78b-t42sc" Oct 30 07:04:25.041603 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:25.041549 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eea3e53-4722-493d-8820-27d28c7e5270-metrics-certs\") pod \"maas-api-f785bd78b-t42sc\" (UID: \"9eea3e53-4722-493d-8820-27d28c7e5270\") " pod="odh-ai-gateway-infra/maas-api-f785bd78b-t42sc" Oct 30 07:04:25.042031 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:25.042008 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eea3e53-4722-493d-8820-27d28c7e5270-openshift-service-ca\") pod \"maas-api-f785bd78b-t42sc\" (UID: \"9eea3e53-4722-493d-8820-27d28c7e5270\") " pod="odh-ai-gateway-infra/maas-api-f785bd78b-t42sc" Oct 30 07:04:25.043891 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:25.043869 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eea3e53-4722-493d-8820-27d28c7e5270-metrics-certs\") pod \"maas-api-f785bd78b-t42sc\" (UID: \"9eea3e53-4722-493d-8820-27d28c7e5270\") " pod="odh-ai-gateway-infra/maas-api-f785bd78b-t42sc" Oct 30 07:04:25.043891 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:25.043882 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9eea3e53-4722-493d-8820-27d28c7e5270-maas-api-tls\") pod \"maas-api-f785bd78b-t42sc\" (UID: \"9eea3e53-4722-493d-8820-27d28c7e5270\") " pod="odh-ai-gateway-infra/maas-api-f785bd78b-t42sc" Oct 30 07:04:25.063709 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:25.063674 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l47wn\" (UniqueName: \"kubernetes.io/projected/9eea3e53-4722-493d-8820-27d28c7e5270-kube-api-access-l47wn\") pod \"maas-api-f785bd78b-t42sc\" (UID: \"9eea3e53-4722-493d-8820-27d28c7e5270\") " pod="odh-ai-gateway-infra/maas-api-f785bd78b-t42sc" Oct 30 07:04:25.172726 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:25.172630 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f785bd78b-t42sc" Oct 30 07:04:25.471341 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:25.471313 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f785bd78b-t42sc"] Oct 30 07:04:25.473869 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:04:25.473807 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eea3e53_4722_493d_8820_27d28c7e5270.slice/crio-f8da3fd5a0293ab3dd0c9feb38304af8e2c85146347413443f9f2057f7153e59 WatchSource:0}: Error finding container f8da3fd5a0293ab3dd0c9feb38304af8e2c85146347413443f9f2057f7153e59: Status 404 returned error can't find the container with id f8da3fd5a0293ab3dd0c9feb38304af8e2c85146347413443f9f2057f7153e59 Oct 30 07:04:25.662840 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:25.662804 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f785bd78b-t42sc" event={"ID":"9eea3e53-4722-493d-8820-27d28c7e5270","Type":"ContainerStarted","Data":"f8da3fd5a0293ab3dd0c9feb38304af8e2c85146347413443f9f2057f7153e59"} Oct 30 07:04:26.668799 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:26.668757 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f785bd78b-t42sc" event={"ID":"9eea3e53-4722-493d-8820-27d28c7e5270","Type":"ContainerStarted","Data":"343978aec378af367e93eaa1a46f126c5b4d804ec5df1990a50f5bf24986218b"} Oct 30 07:04:26.669241 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:26.668857 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-f785bd78b-t42sc" Oct 30 07:04:26.707758 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:26.707712 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-f785bd78b-t42sc" podStartSLOduration=2.346284551 podStartE2EDuration="2.7076995s" totalImagesPullingTime="361.414949ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:04:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:04:26.707017678 +0000 UTC m=+473.050065644" watchObservedRunningTime="2026-10-30 07:04:26.7076995 +0000 UTC m=+473.050747459" Oct 30 07:04:32.679462 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:32.679426 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-f785bd78b-t42sc" Oct 30 07:04:32.748070 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:32.748037 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8b74c94db-jqz4t"] Oct 30 07:04:32.748459 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:32.748432 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-8b74c94db-jqz4t" podUID="6b040c64-4330-4e22-a88b-fa7ab9eac634" containerName="maas-api" containerID="cri-o://9bd70a55f199e3169946b2a2b361a161bd01db00a46b7365e393d45275b8a8a5" gracePeriod=30 Oct 30 07:04:32.989697 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:32.989671 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8b74c94db-jqz4t" Oct 30 07:04:33.106113 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:33.106070 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b040c64-4330-4e22-a88b-fa7ab9eac634-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b040c64-4330-4e22-a88b-fa7ab9eac634-metrics-certs\") pod \"6b040c64-4330-4e22-a88b-fa7ab9eac634\" (UID: \"6b040c64-4330-4e22-a88b-fa7ab9eac634\") " Oct 30 07:04:33.106317 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:33.106123 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b040c64-4330-4e22-a88b-fa7ab9eac634-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b040c64-4330-4e22-a88b-fa7ab9eac634-openshift-service-ca\") pod \"6b040c64-4330-4e22-a88b-fa7ab9eac634\" (UID: \"6b040c64-4330-4e22-a88b-fa7ab9eac634\") " Oct 30 07:04:33.106317 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:33.106177 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b040c64-4330-4e22-a88b-fa7ab9eac634-kube-api-access-tgg9j\" (UniqueName: \"kubernetes.io/projected/6b040c64-4330-4e22-a88b-fa7ab9eac634-kube-api-access-tgg9j\") pod \"6b040c64-4330-4e22-a88b-fa7ab9eac634\" (UID: \"6b040c64-4330-4e22-a88b-fa7ab9eac634\") " Oct 30 07:04:33.106317 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:33.106204 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b040c64-4330-4e22-a88b-fa7ab9eac634-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b040c64-4330-4e22-a88b-fa7ab9eac634-maas-api-tls\") pod \"6b040c64-4330-4e22-a88b-fa7ab9eac634\" (UID: \"6b040c64-4330-4e22-a88b-fa7ab9eac634\") " Oct 30 07:04:33.106608 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:33.106552 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b040c64-4330-4e22-a88b-fa7ab9eac634-openshift-service-ca" pod "6b040c64-4330-4e22-a88b-fa7ab9eac634" (UID: "6b040c64-4330-4e22-a88b-fa7ab9eac634"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:04:33.108490 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:33.108452 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b040c64-4330-4e22-a88b-fa7ab9eac634-maas-api-tls" pod "6b040c64-4330-4e22-a88b-fa7ab9eac634" (UID: "6b040c64-4330-4e22-a88b-fa7ab9eac634"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:04:33.108634 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:33.108512 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b040c64-4330-4e22-a88b-fa7ab9eac634-kube-api-access-tgg9j" pod "6b040c64-4330-4e22-a88b-fa7ab9eac634" (UID: "6b040c64-4330-4e22-a88b-fa7ab9eac634"). InnerVolumeSpecName "kube-api-access-tgg9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:04:33.108634 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:33.108526 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b040c64-4330-4e22-a88b-fa7ab9eac634-metrics-certs" pod "6b040c64-4330-4e22-a88b-fa7ab9eac634" (UID: "6b040c64-4330-4e22-a88b-fa7ab9eac634"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:04:33.207401 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:33.207362 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgg9j\" (UniqueName: \"kubernetes.io/projected/6b040c64-4330-4e22-a88b-fa7ab9eac634-kube-api-access-tgg9j\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:04:33.207401 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:33.207394 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b040c64-4330-4e22-a88b-fa7ab9eac634-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:04:33.207401 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:33.207407 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b040c64-4330-4e22-a88b-fa7ab9eac634-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:04:33.207676 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:33.207415 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b040c64-4330-4e22-a88b-fa7ab9eac634-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:04:33.698768 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:33.698733 4951 generic.go:356] "Generic (PLEG): container finished" podID="6b040c64-4330-4e22-a88b-fa7ab9eac634" containerID="9bd70a55f199e3169946b2a2b361a161bd01db00a46b7365e393d45275b8a8a5" exitCode=0 Oct 30 07:04:33.699161 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:33.698780 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8b74c94db-jqz4t" Oct 30 07:04:33.699161 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:33.698780 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8b74c94db-jqz4t" event={"ID":"6b040c64-4330-4e22-a88b-fa7ab9eac634","Type":"ContainerDied","Data":"9bd70a55f199e3169946b2a2b361a161bd01db00a46b7365e393d45275b8a8a5"} Oct 30 07:04:33.699161 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:33.698911 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8b74c94db-jqz4t" event={"ID":"6b040c64-4330-4e22-a88b-fa7ab9eac634","Type":"ContainerDied","Data":"ecc59e72d71d8aa8039f30d56adbf0337b4d29b715be2bd7fb0cec5196ba0647"} Oct 30 07:04:33.699161 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:33.698941 4951 scope.go:122] "RemoveContainer" containerID="9bd70a55f199e3169946b2a2b361a161bd01db00a46b7365e393d45275b8a8a5" Oct 30 07:04:33.708217 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:33.708190 4951 scope.go:122] "RemoveContainer" containerID="9bd70a55f199e3169946b2a2b361a161bd01db00a46b7365e393d45275b8a8a5" Oct 30 07:04:33.708506 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:04:33.708486 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bd70a55f199e3169946b2a2b361a161bd01db00a46b7365e393d45275b8a8a5\": container with ID starting with 9bd70a55f199e3169946b2a2b361a161bd01db00a46b7365e393d45275b8a8a5 not found: ID does not exist" containerID="9bd70a55f199e3169946b2a2b361a161bd01db00a46b7365e393d45275b8a8a5" Oct 30 07:04:33.708566 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:33.708516 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bd70a55f199e3169946b2a2b361a161bd01db00a46b7365e393d45275b8a8a5"} err="failed to get container status \"9bd70a55f199e3169946b2a2b361a161bd01db00a46b7365e393d45275b8a8a5\": rpc error: code = NotFound desc = could not find container \"9bd70a55f199e3169946b2a2b361a161bd01db00a46b7365e393d45275b8a8a5\": container with ID starting with 9bd70a55f199e3169946b2a2b361a161bd01db00a46b7365e393d45275b8a8a5 not found: ID does not exist" Oct 30 07:04:33.745890 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:33.745858 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8b74c94db-jqz4t"] Oct 30 07:04:33.756802 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:33.756770 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-8b74c94db-jqz4t"] Oct 30 07:04:33.769731 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:04:33.769680 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b040c64-4330-4e22-a88b-fa7ab9eac634" path="/var/lib/kubelet/pods/6b040c64-4330-4e22-a88b-fa7ab9eac634/volumes" Oct 30 07:04:53.022258 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:04:53.022216 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:05:23.024991 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:05:23.024950 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:05:43.068615 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:43.068566 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-ml7hg"] Oct 30 07:05:43.069162 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:43.069034 4951 cpu_manager.go:414] "containerMap: removing container" podUID="6b040c64-4330-4e22-a88b-fa7ab9eac634" containerName="maas-api" Oct 30 07:05:43.069162 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:43.069050 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b040c64-4330-4e22-a88b-fa7ab9eac634" containerName="maas-api" Oct 30 07:05:43.069162 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:43.069131 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b040c64-4330-4e22-a88b-fa7ab9eac634" containerName="maas-api" Oct 30 07:05:43.072233 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:43.072207 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-ml7hg" Oct 30 07:05:43.096817 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:43.096789 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 07:05:43.096961 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:43.096871 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 07:05:43.096961 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:43.096918 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 07:05:43.096961 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:43.096899 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 07:05:43.096961 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:43.096927 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-qdnck\"" Oct 30 07:05:43.098403 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:43.098382 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-ml7hg"] Oct 30 07:05:43.196396 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:43.196361 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf8st\" (UniqueName: \"kubernetes.io/projected/a58e5361-6a36-4bed-8a42-17dd6b0baf38-kube-api-access-qf8st\") pod \"ai-gateway-controller-7d85675f98-ml7hg\" (UID: \"a58e5361-6a36-4bed-8a42-17dd6b0baf38\") " pod="opendatahub/ai-gateway-controller-7d85675f98-ml7hg" Oct 30 07:05:43.196396 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:43.196399 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a58e5361-6a36-4bed-8a42-17dd6b0baf38-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-ml7hg\" (UID: \"a58e5361-6a36-4bed-8a42-17dd6b0baf38\") " pod="opendatahub/ai-gateway-controller-7d85675f98-ml7hg" Oct 30 07:05:43.297390 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:43.297347 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf8st\" (UniqueName: \"kubernetes.io/projected/a58e5361-6a36-4bed-8a42-17dd6b0baf38-kube-api-access-qf8st\") pod \"ai-gateway-controller-7d85675f98-ml7hg\" (UID: \"a58e5361-6a36-4bed-8a42-17dd6b0baf38\") " pod="opendatahub/ai-gateway-controller-7d85675f98-ml7hg" Oct 30 07:05:43.297611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:43.297395 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a58e5361-6a36-4bed-8a42-17dd6b0baf38-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-ml7hg\" (UID: \"a58e5361-6a36-4bed-8a42-17dd6b0baf38\") " pod="opendatahub/ai-gateway-controller-7d85675f98-ml7hg" Oct 30 07:05:43.299888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:43.299868 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a58e5361-6a36-4bed-8a42-17dd6b0baf38-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-ml7hg\" (UID: \"a58e5361-6a36-4bed-8a42-17dd6b0baf38\") " pod="opendatahub/ai-gateway-controller-7d85675f98-ml7hg" Oct 30 07:05:43.320825 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:43.320772 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf8st\" (UniqueName: \"kubernetes.io/projected/a58e5361-6a36-4bed-8a42-17dd6b0baf38-kube-api-access-qf8st\") pod \"ai-gateway-controller-7d85675f98-ml7hg\" (UID: \"a58e5361-6a36-4bed-8a42-17dd6b0baf38\") " pod="opendatahub/ai-gateway-controller-7d85675f98-ml7hg" Oct 30 07:05:43.384339 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:43.384300 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-ml7hg" Oct 30 07:05:43.676379 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:43.676354 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-ml7hg"] Oct 30 07:05:43.678807 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:05:43.678772 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda58e5361_6a36_4bed_8a42_17dd6b0baf38.slice/crio-088d0348cb07d2fc4f556fc6053a39ff9d1b24ab899fac31cbfb8a52e51615db WatchSource:0}: Error finding container 088d0348cb07d2fc4f556fc6053a39ff9d1b24ab899fac31cbfb8a52e51615db: Status 404 returned error can't find the container with id 088d0348cb07d2fc4f556fc6053a39ff9d1b24ab899fac31cbfb8a52e51615db Oct 30 07:05:43.977964 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:43.977883 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-ml7hg" event={"ID":"a58e5361-6a36-4bed-8a42-17dd6b0baf38","Type":"ContainerStarted","Data":"088d0348cb07d2fc4f556fc6053a39ff9d1b24ab899fac31cbfb8a52e51615db"} Oct 30 07:05:45.988370 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:45.988334 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-ml7hg" event={"ID":"a58e5361-6a36-4bed-8a42-17dd6b0baf38","Type":"ContainerStarted","Data":"98fd81bd0871da8ac53e49ff5f93145c99fbe5c59fc69b6f179f256ebca90574"} Oct 30 07:05:45.988918 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:45.988451 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-ml7hg" Oct 30 07:05:46.027799 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:46.027744 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-ml7hg" podStartSLOduration=1.596532355 podStartE2EDuration="3.027728889s" totalImagesPullingTime="1.431196534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:05:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:05:46.026843927 +0000 UTC m=+552.369891912" watchObservedRunningTime="2026-10-30 07:05:46.027728889 +0000 UTC m=+552.370776854" Oct 30 07:05:47.550881 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:47.550844 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-m4l2r"] Oct 30 07:05:47.554499 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:47.554479 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-m4l2r" Oct 30 07:05:47.571118 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:47.571090 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 07:05:47.582330 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:47.582302 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-m4shx\"" Oct 30 07:05:47.583677 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:47.583654 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-m4l2r"] Oct 30 07:05:47.640036 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:47.640003 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-pbml8"] Oct 30 07:05:47.643288 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:47.643270 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-pbml8" Oct 30 07:05:47.664833 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:47.664799 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-pbml8"] Oct 30 07:05:47.734369 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:47.734336 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmhhm\" (UniqueName: \"kubernetes.io/projected/c834492c-b73f-41d5-b3c0-95ad02f5e005-kube-api-access-xmhhm\") pod \"payload-pre-processing-7b98bb678c-m4l2r\" (UID: \"c834492c-b73f-41d5-b3c0-95ad02f5e005\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-m4l2r" Oct 30 07:05:47.734548 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:47.734383 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/861854e2-21ba-40a6-b1b5-e3e36bd0858f-plugins-config-volume\") pod \"payload-processing-5f78687d5b-pbml8\" (UID: \"861854e2-21ba-40a6-b1b5-e3e36bd0858f\") " pod="openshift-ingress/payload-processing-5f78687d5b-pbml8" Oct 30 07:05:47.734548 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:47.734487 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c834492c-b73f-41d5-b3c0-95ad02f5e005-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-m4l2r\" (UID: \"c834492c-b73f-41d5-b3c0-95ad02f5e005\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-m4l2r" Oct 30 07:05:47.734548 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:47.734539 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98xr9\" (UniqueName: \"kubernetes.io/projected/861854e2-21ba-40a6-b1b5-e3e36bd0858f-kube-api-access-98xr9\") pod \"payload-processing-5f78687d5b-pbml8\" (UID: \"861854e2-21ba-40a6-b1b5-e3e36bd0858f\") " pod="openshift-ingress/payload-processing-5f78687d5b-pbml8" Oct 30 07:05:47.835136 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:47.835049 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/861854e2-21ba-40a6-b1b5-e3e36bd0858f-plugins-config-volume\") pod \"payload-processing-5f78687d5b-pbml8\" (UID: \"861854e2-21ba-40a6-b1b5-e3e36bd0858f\") " pod="openshift-ingress/payload-processing-5f78687d5b-pbml8" Oct 30 07:05:47.835136 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:47.835117 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c834492c-b73f-41d5-b3c0-95ad02f5e005-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-m4l2r\" (UID: \"c834492c-b73f-41d5-b3c0-95ad02f5e005\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-m4l2r" Oct 30 07:05:47.835354 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:47.835147 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98xr9\" (UniqueName: \"kubernetes.io/projected/861854e2-21ba-40a6-b1b5-e3e36bd0858f-kube-api-access-98xr9\") pod \"payload-processing-5f78687d5b-pbml8\" (UID: \"861854e2-21ba-40a6-b1b5-e3e36bd0858f\") " pod="openshift-ingress/payload-processing-5f78687d5b-pbml8" Oct 30 07:05:47.835354 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:47.835197 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmhhm\" (UniqueName: \"kubernetes.io/projected/c834492c-b73f-41d5-b3c0-95ad02f5e005-kube-api-access-xmhhm\") pod \"payload-pre-processing-7b98bb678c-m4l2r\" (UID: \"c834492c-b73f-41d5-b3c0-95ad02f5e005\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-m4l2r" Oct 30 07:05:47.835836 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:47.835815 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/861854e2-21ba-40a6-b1b5-e3e36bd0858f-plugins-config-volume\") pod \"payload-processing-5f78687d5b-pbml8\" (UID: \"861854e2-21ba-40a6-b1b5-e3e36bd0858f\") " pod="openshift-ingress/payload-processing-5f78687d5b-pbml8" Oct 30 07:05:47.835900 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:47.835882 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c834492c-b73f-41d5-b3c0-95ad02f5e005-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-m4l2r\" (UID: \"c834492c-b73f-41d5-b3c0-95ad02f5e005\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-m4l2r" Oct 30 07:05:47.857356 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:47.857330 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98xr9\" (UniqueName: \"kubernetes.io/projected/861854e2-21ba-40a6-b1b5-e3e36bd0858f-kube-api-access-98xr9\") pod \"payload-processing-5f78687d5b-pbml8\" (UID: \"861854e2-21ba-40a6-b1b5-e3e36bd0858f\") " pod="openshift-ingress/payload-processing-5f78687d5b-pbml8" Oct 30 07:05:47.857557 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:47.857537 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmhhm\" (UniqueName: \"kubernetes.io/projected/c834492c-b73f-41d5-b3c0-95ad02f5e005-kube-api-access-xmhhm\") pod \"payload-pre-processing-7b98bb678c-m4l2r\" (UID: \"c834492c-b73f-41d5-b3c0-95ad02f5e005\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-m4l2r" Oct 30 07:05:47.868193 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:47.868171 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-m4l2r" Oct 30 07:05:47.955220 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:47.955184 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-pbml8" Oct 30 07:05:48.158220 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:48.158191 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-m4l2r"] Oct 30 07:05:48.160693 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:05:48.160665 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc834492c_b73f_41d5_b3c0_95ad02f5e005.slice/crio-30fdc210ddd9aa2dcf309fe10fc3c157427d484153841ce72f4005b79f3910c3 WatchSource:0}: Error finding container 30fdc210ddd9aa2dcf309fe10fc3c157427d484153841ce72f4005b79f3910c3: Status 404 returned error can't find the container with id 30fdc210ddd9aa2dcf309fe10fc3c157427d484153841ce72f4005b79f3910c3 Oct 30 07:05:48.250037 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:48.250016 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-pbml8"] Oct 30 07:05:48.251808 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:05:48.251773 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod861854e2_21ba_40a6_b1b5_e3e36bd0858f.slice/crio-5bd2ab677458346d2d9e3e550e0a32a21853c9c06f77329e4c3a7d67aacf61fe WatchSource:0}: Error finding container 5bd2ab677458346d2d9e3e550e0a32a21853c9c06f77329e4c3a7d67aacf61fe: Status 404 returned error can't find the container with id 5bd2ab677458346d2d9e3e550e0a32a21853c9c06f77329e4c3a7d67aacf61fe Oct 30 07:05:49.005784 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:49.005735 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-pbml8" event={"ID":"861854e2-21ba-40a6-b1b5-e3e36bd0858f","Type":"ContainerStarted","Data":"5bd2ab677458346d2d9e3e550e0a32a21853c9c06f77329e4c3a7d67aacf61fe"} Oct 30 07:05:49.007310 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:49.007278 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-m4l2r" event={"ID":"c834492c-b73f-41d5-b3c0-95ad02f5e005","Type":"ContainerStarted","Data":"30fdc210ddd9aa2dcf309fe10fc3c157427d484153841ce72f4005b79f3910c3"} Oct 30 07:05:51.018718 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:51.018674 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-pbml8" event={"ID":"861854e2-21ba-40a6-b1b5-e3e36bd0858f","Type":"ContainerStarted","Data":"1b85932e39076d004bc8015015ba10f0c9e80161f1c135948ab4928d3c2098e9"} Oct 30 07:05:51.019165 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:51.018844 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-pbml8" Oct 30 07:05:51.020283 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:51.020254 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-m4l2r" event={"ID":"c834492c-b73f-41d5-b3c0-95ad02f5e005","Type":"ContainerStarted","Data":"25a7be151f4cc88ec260b75ad78e8b787916ec5c4a3f9b29edd89d6da5be634a"} Oct 30 07:05:51.020396 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:51.020311 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-m4l2r" Oct 30 07:05:51.057055 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:51.057007 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-pbml8" podStartSLOduration=1.882708952 podStartE2EDuration="4.056993849s" totalImagesPullingTime="2.174284897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:05:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:05:51.056664621 +0000 UTC m=+557.399712587" watchObservedRunningTime="2026-10-30 07:05:51.056993849 +0000 UTC m=+557.400041814" Oct 30 07:05:51.095027 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:51.094978 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-m4l2r" podStartSLOduration=1.924829022 podStartE2EDuration="4.094964862s" totalImagesPullingTime="2.17013584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:05:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:05:51.094707059 +0000 UTC m=+557.437755045" watchObservedRunningTime="2026-10-30 07:05:51.094964862 +0000 UTC m=+557.438012826" Oct 30 07:05:53.022652 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:05:53.022611 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:05:56.995287 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:56.995254 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-ml7hg" Oct 30 07:05:57.301487 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:57.301455 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-795446ff4c-nbl2l"] Oct 30 07:05:57.305266 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:57.305250 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-795446ff4c-nbl2l" Oct 30 07:05:57.314612 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:57.314581 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2tmm\" (UniqueName: \"kubernetes.io/projected/483fcce9-07d5-4224-bfc9-6ae15c087276-kube-api-access-z2tmm\") pod \"ai-gateway-controller-795446ff4c-nbl2l\" (UID: \"483fcce9-07d5-4224-bfc9-6ae15c087276\") " pod="opendatahub/ai-gateway-controller-795446ff4c-nbl2l" Oct 30 07:05:57.314755 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:57.314647 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/483fcce9-07d5-4224-bfc9-6ae15c087276-metrics-certs\") pod \"ai-gateway-controller-795446ff4c-nbl2l\" (UID: \"483fcce9-07d5-4224-bfc9-6ae15c087276\") " pod="opendatahub/ai-gateway-controller-795446ff4c-nbl2l" Oct 30 07:05:57.325725 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:57.325697 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-795446ff4c-nbl2l"] Oct 30 07:05:57.415890 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:57.415851 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2tmm\" (UniqueName: \"kubernetes.io/projected/483fcce9-07d5-4224-bfc9-6ae15c087276-kube-api-access-z2tmm\") pod \"ai-gateway-controller-795446ff4c-nbl2l\" (UID: \"483fcce9-07d5-4224-bfc9-6ae15c087276\") " pod="opendatahub/ai-gateway-controller-795446ff4c-nbl2l" Oct 30 07:05:57.416077 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:57.415909 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/483fcce9-07d5-4224-bfc9-6ae15c087276-metrics-certs\") pod \"ai-gateway-controller-795446ff4c-nbl2l\" (UID: \"483fcce9-07d5-4224-bfc9-6ae15c087276\") " pod="opendatahub/ai-gateway-controller-795446ff4c-nbl2l" Oct 30 07:05:57.418313 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:57.418283 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/483fcce9-07d5-4224-bfc9-6ae15c087276-metrics-certs\") pod \"ai-gateway-controller-795446ff4c-nbl2l\" (UID: \"483fcce9-07d5-4224-bfc9-6ae15c087276\") " pod="opendatahub/ai-gateway-controller-795446ff4c-nbl2l" Oct 30 07:05:57.442740 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:57.442718 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2tmm\" (UniqueName: \"kubernetes.io/projected/483fcce9-07d5-4224-bfc9-6ae15c087276-kube-api-access-z2tmm\") pod \"ai-gateway-controller-795446ff4c-nbl2l\" (UID: \"483fcce9-07d5-4224-bfc9-6ae15c087276\") " pod="opendatahub/ai-gateway-controller-795446ff4c-nbl2l" Oct 30 07:05:57.618064 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:57.617977 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-795446ff4c-nbl2l" Oct 30 07:05:57.910618 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:57.910004 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-795446ff4c-nbl2l"] Oct 30 07:05:57.912235 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:05:57.912205 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod483fcce9_07d5_4224_bfc9_6ae15c087276.slice/crio-3932589a213a34defba0052cb9b59274d5c8b8c7ad8148b1e54fddca33188281 WatchSource:0}: Error finding container 3932589a213a34defba0052cb9b59274d5c8b8c7ad8148b1e54fddca33188281: Status 404 returned error can't find the container with id 3932589a213a34defba0052cb9b59274d5c8b8c7ad8148b1e54fddca33188281 Oct 30 07:05:58.049652 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:58.049609 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-795446ff4c-nbl2l" event={"ID":"483fcce9-07d5-4224-bfc9-6ae15c087276","Type":"ContainerStarted","Data":"3932589a213a34defba0052cb9b59274d5c8b8c7ad8148b1e54fddca33188281"} Oct 30 07:05:59.056385 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:59.056345 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-795446ff4c-nbl2l" event={"ID":"483fcce9-07d5-4224-bfc9-6ae15c087276","Type":"ContainerStarted","Data":"a5ad3df83bc807cd8f345b5ec6cbab738d06b3db4b8d784e43684e3bbefeb740"} Oct 30 07:05:59.056893 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:59.056440 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-795446ff4c-nbl2l" Oct 30 07:05:59.095768 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:05:59.095708 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-795446ff4c-nbl2l" podStartSLOduration=1.7822104730000001 podStartE2EDuration="2.095688563s" totalImagesPullingTime="313.47809ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:05:59.094418634 +0000 UTC m=+565.437466625" watchObservedRunningTime="2026-10-30 07:05:59.095688563 +0000 UTC m=+565.438736529" Oct 30 07:06:02.027887 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:02.027861 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-pbml8" Oct 30 07:06:02.028390 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:02.027935 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-m4l2r" Oct 30 07:06:10.062855 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:10.062827 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-795446ff4c-nbl2l" Oct 30 07:06:10.130813 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:10.130781 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-ml7hg"] Oct 30 07:06:10.131031 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:10.131004 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-ml7hg" podUID="a58e5361-6a36-4bed-8a42-17dd6b0baf38" containerName="manager" containerID="cri-o://98fd81bd0871da8ac53e49ff5f93145c99fbe5c59fc69b6f179f256ebca90574" gracePeriod=30 Oct 30 07:06:10.372612 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:10.372553 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-ml7hg" Oct 30 07:06:10.412779 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:10.412752 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a58e5361-6a36-4bed-8a42-17dd6b0baf38-kube-api-access-qf8st\" (UniqueName: \"kubernetes.io/projected/a58e5361-6a36-4bed-8a42-17dd6b0baf38-kube-api-access-qf8st\") pod \"a58e5361-6a36-4bed-8a42-17dd6b0baf38\" (UID: \"a58e5361-6a36-4bed-8a42-17dd6b0baf38\") " Oct 30 07:06:10.412893 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:10.412826 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a58e5361-6a36-4bed-8a42-17dd6b0baf38-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a58e5361-6a36-4bed-8a42-17dd6b0baf38-metrics-certs\") pod \"a58e5361-6a36-4bed-8a42-17dd6b0baf38\" (UID: \"a58e5361-6a36-4bed-8a42-17dd6b0baf38\") " Oct 30 07:06:10.414882 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:10.414858 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a58e5361-6a36-4bed-8a42-17dd6b0baf38-kube-api-access-qf8st" pod "a58e5361-6a36-4bed-8a42-17dd6b0baf38" (UID: "a58e5361-6a36-4bed-8a42-17dd6b0baf38"). InnerVolumeSpecName "kube-api-access-qf8st". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:06:10.414990 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:10.414967 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a58e5361-6a36-4bed-8a42-17dd6b0baf38-metrics-certs" pod "a58e5361-6a36-4bed-8a42-17dd6b0baf38" (UID: "a58e5361-6a36-4bed-8a42-17dd6b0baf38"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:06:10.513895 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:10.513855 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a58e5361-6a36-4bed-8a42-17dd6b0baf38-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:06:10.513895 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:10.513891 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf8st\" (UniqueName: \"kubernetes.io/projected/a58e5361-6a36-4bed-8a42-17dd6b0baf38-kube-api-access-qf8st\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:06:11.107836 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:11.107801 4951 generic.go:356] "Generic (PLEG): container finished" podID="a58e5361-6a36-4bed-8a42-17dd6b0baf38" containerID="98fd81bd0871da8ac53e49ff5f93145c99fbe5c59fc69b6f179f256ebca90574" exitCode=0 Oct 30 07:06:11.108247 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:11.107846 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-ml7hg" Oct 30 07:06:11.108247 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:11.107885 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-ml7hg" event={"ID":"a58e5361-6a36-4bed-8a42-17dd6b0baf38","Type":"ContainerDied","Data":"98fd81bd0871da8ac53e49ff5f93145c99fbe5c59fc69b6f179f256ebca90574"} Oct 30 07:06:11.108247 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:11.107937 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-ml7hg" event={"ID":"a58e5361-6a36-4bed-8a42-17dd6b0baf38","Type":"ContainerDied","Data":"088d0348cb07d2fc4f556fc6053a39ff9d1b24ab899fac31cbfb8a52e51615db"} Oct 30 07:06:11.108247 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:11.107962 4951 scope.go:122] "RemoveContainer" containerID="98fd81bd0871da8ac53e49ff5f93145c99fbe5c59fc69b6f179f256ebca90574" Oct 30 07:06:11.117401 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:11.117384 4951 scope.go:122] "RemoveContainer" containerID="98fd81bd0871da8ac53e49ff5f93145c99fbe5c59fc69b6f179f256ebca90574" Oct 30 07:06:11.117733 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:06:11.117711 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98fd81bd0871da8ac53e49ff5f93145c99fbe5c59fc69b6f179f256ebca90574\": container with ID starting with 98fd81bd0871da8ac53e49ff5f93145c99fbe5c59fc69b6f179f256ebca90574 not found: ID does not exist" containerID="98fd81bd0871da8ac53e49ff5f93145c99fbe5c59fc69b6f179f256ebca90574" Oct 30 07:06:11.117786 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:11.117744 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98fd81bd0871da8ac53e49ff5f93145c99fbe5c59fc69b6f179f256ebca90574"} err="failed to get container status \"98fd81bd0871da8ac53e49ff5f93145c99fbe5c59fc69b6f179f256ebca90574\": rpc error: code = NotFound desc = could not find container \"98fd81bd0871da8ac53e49ff5f93145c99fbe5c59fc69b6f179f256ebca90574\": container with ID starting with 98fd81bd0871da8ac53e49ff5f93145c99fbe5c59fc69b6f179f256ebca90574 not found: ID does not exist" Oct 30 07:06:11.155624 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:11.155590 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-ml7hg"] Oct 30 07:06:11.166565 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:11.166530 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-ml7hg"] Oct 30 07:06:11.769707 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:11.769675 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a58e5361-6a36-4bed-8a42-17dd6b0baf38" path="/var/lib/kubelet/pods/a58e5361-6a36-4bed-8a42-17dd6b0baf38/volumes" Oct 30 07:06:23.023086 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:06:23.022972 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:06:32.882848 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:32.882807 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-m8ksd"] Oct 30 07:06:32.883302 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:32.883284 4951 cpu_manager.go:414] "containerMap: removing container" podUID="a58e5361-6a36-4bed-8a42-17dd6b0baf38" containerName="manager" Oct 30 07:06:32.883366 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:32.883302 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a58e5361-6a36-4bed-8a42-17dd6b0baf38" containerName="manager" Oct 30 07:06:32.883407 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:32.883374 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="a58e5361-6a36-4bed-8a42-17dd6b0baf38" containerName="manager" Oct 30 07:06:32.888915 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:32.888893 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8ksd" Oct 30 07:06:32.905307 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:32.905278 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4cvnh\"" Oct 30 07:06:32.916426 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:32.916401 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 07:06:32.916554 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:32.916434 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 07:06:32.918174 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:32.918153 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-m8ksd"] Oct 30 07:06:32.922325 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:32.922294 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 07:06:33.005301 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:33.005272 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbsdb\" (UniqueName: \"kubernetes.io/projected/c9b6e389-2e4a-45d1-a3cf-a15d930ebb63-kube-api-access-kbsdb\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8ksd\" (UID: \"c9b6e389-2e4a-45d1-a3cf-a15d930ebb63\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8ksd" Oct 30 07:06:33.005477 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:33.005306 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9b6e389-2e4a-45d1-a3cf-a15d930ebb63-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8ksd\" (UID: \"c9b6e389-2e4a-45d1-a3cf-a15d930ebb63\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8ksd" Oct 30 07:06:33.005477 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:33.005330 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9b6e389-2e4a-45d1-a3cf-a15d930ebb63-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8ksd\" (UID: \"c9b6e389-2e4a-45d1-a3cf-a15d930ebb63\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8ksd" Oct 30 07:06:33.005477 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:33.005387 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9b6e389-2e4a-45d1-a3cf-a15d930ebb63-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8ksd\" (UID: \"c9b6e389-2e4a-45d1-a3cf-a15d930ebb63\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8ksd" Oct 30 07:06:33.005477 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:33.005470 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9b6e389-2e4a-45d1-a3cf-a15d930ebb63-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8ksd\" (UID: \"c9b6e389-2e4a-45d1-a3cf-a15d930ebb63\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8ksd" Oct 30 07:06:33.005674 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:33.005494 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9b6e389-2e4a-45d1-a3cf-a15d930ebb63-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8ksd\" (UID: \"c9b6e389-2e4a-45d1-a3cf-a15d930ebb63\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8ksd" Oct 30 07:06:33.106064 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:33.106026 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbsdb\" (UniqueName: \"kubernetes.io/projected/c9b6e389-2e4a-45d1-a3cf-a15d930ebb63-kube-api-access-kbsdb\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8ksd\" (UID: \"c9b6e389-2e4a-45d1-a3cf-a15d930ebb63\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8ksd" Oct 30 07:06:33.106064 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:33.106067 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9b6e389-2e4a-45d1-a3cf-a15d930ebb63-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8ksd\" (UID: \"c9b6e389-2e4a-45d1-a3cf-a15d930ebb63\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8ksd" Oct 30 07:06:33.106331 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:33.106097 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9b6e389-2e4a-45d1-a3cf-a15d930ebb63-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8ksd\" (UID: \"c9b6e389-2e4a-45d1-a3cf-a15d930ebb63\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8ksd" Oct 30 07:06:33.106331 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:33.106131 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9b6e389-2e4a-45d1-a3cf-a15d930ebb63-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8ksd\" (UID: \"c9b6e389-2e4a-45d1-a3cf-a15d930ebb63\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8ksd" Oct 30 07:06:33.106331 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:33.106199 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9b6e389-2e4a-45d1-a3cf-a15d930ebb63-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8ksd\" (UID: \"c9b6e389-2e4a-45d1-a3cf-a15d930ebb63\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8ksd" Oct 30 07:06:33.106331 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:33.106224 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9b6e389-2e4a-45d1-a3cf-a15d930ebb63-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8ksd\" (UID: \"c9b6e389-2e4a-45d1-a3cf-a15d930ebb63\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8ksd" Oct 30 07:06:33.106543 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:33.106455 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9b6e389-2e4a-45d1-a3cf-a15d930ebb63-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8ksd\" (UID: \"c9b6e389-2e4a-45d1-a3cf-a15d930ebb63\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8ksd" Oct 30 07:06:33.106543 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:33.106533 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9b6e389-2e4a-45d1-a3cf-a15d930ebb63-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8ksd\" (UID: \"c9b6e389-2e4a-45d1-a3cf-a15d930ebb63\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8ksd" Oct 30 07:06:33.106663 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:33.106635 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9b6e389-2e4a-45d1-a3cf-a15d930ebb63-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8ksd\" (UID: \"c9b6e389-2e4a-45d1-a3cf-a15d930ebb63\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8ksd" Oct 30 07:06:33.108384 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:33.108362 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9b6e389-2e4a-45d1-a3cf-a15d930ebb63-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8ksd\" (UID: \"c9b6e389-2e4a-45d1-a3cf-a15d930ebb63\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8ksd" Oct 30 07:06:33.108805 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:33.108785 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9b6e389-2e4a-45d1-a3cf-a15d930ebb63-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8ksd\" (UID: \"c9b6e389-2e4a-45d1-a3cf-a15d930ebb63\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8ksd" Oct 30 07:06:33.142511 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:33.142449 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbsdb\" (UniqueName: \"kubernetes.io/projected/c9b6e389-2e4a-45d1-a3cf-a15d930ebb63-kube-api-access-kbsdb\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m8ksd\" (UID: \"c9b6e389-2e4a-45d1-a3cf-a15d930ebb63\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8ksd" Oct 30 07:06:33.202002 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:33.201973 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8ksd" Oct 30 07:06:33.496831 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:33.496793 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-m8ksd"] Oct 30 07:06:33.501802 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:06:33.501767 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b6e389_2e4a_45d1_a3cf_a15d930ebb63.slice/crio-53a05d36b8983b9190b09de861dae55bfc3a39e028a6ca34b35157351d452483 WatchSource:0}: Error finding container 53a05d36b8983b9190b09de861dae55bfc3a39e028a6ca34b35157351d452483: Status 404 returned error can't find the container with id 53a05d36b8983b9190b09de861dae55bfc3a39e028a6ca34b35157351d452483 Oct 30 07:06:33.815837 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:33.815801 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/3.log" Oct 30 07:06:33.819192 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:33.819085 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/3.log" Oct 30 07:06:33.823489 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:33.823457 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 07:06:33.827116 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:33.827091 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 07:06:34.211081 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:34.210959 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8ksd" event={"ID":"c9b6e389-2e4a-45d1-a3cf-a15d930ebb63","Type":"ContainerStarted","Data":"53a05d36b8983b9190b09de861dae55bfc3a39e028a6ca34b35157351d452483"} Oct 30 07:06:35.216970 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:35.216930 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8ksd" event={"ID":"c9b6e389-2e4a-45d1-a3cf-a15d930ebb63","Type":"ContainerStarted","Data":"d89d79b545c99b88804b0cf84d5988bd5a7412619d96b176b3ec872b12596b89"} Oct 30 07:06:35.261946 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:35.261888 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8ksd" podStartSLOduration=2.201355559 podStartE2EDuration="3.261871418s" totalImagesPullingTime="1.060515859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:06:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:06:35.25999056 +0000 UTC m=+601.603038554" watchObservedRunningTime="2026-10-30 07:06:35.261871418 +0000 UTC m=+601.604919383" Oct 30 07:06:36.202387 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:36.202353 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8ksd" Oct 30 07:06:36.933037 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:36.933001 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-7dtn7"] Oct 30 07:06:36.942631 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:36.942557 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7dtn7" Oct 30 07:06:36.957656 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:36.957623 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 07:06:36.959484 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:36.959458 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-7dtn7"] Oct 30 07:06:37.041929 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:37.041893 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cb568ba-79ac-405d-af09-bde6c2b47a53-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7dtn7\" (UID: \"2cb568ba-79ac-405d-af09-bde6c2b47a53\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7dtn7" Oct 30 07:06:37.042099 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:37.041940 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cb568ba-79ac-405d-af09-bde6c2b47a53-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7dtn7\" (UID: \"2cb568ba-79ac-405d-af09-bde6c2b47a53\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7dtn7" Oct 30 07:06:37.042099 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:37.042043 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb568ba-79ac-405d-af09-bde6c2b47a53-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7dtn7\" (UID: \"2cb568ba-79ac-405d-af09-bde6c2b47a53\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7dtn7" Oct 30 07:06:37.042186 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:37.042148 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cb568ba-79ac-405d-af09-bde6c2b47a53-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7dtn7\" (UID: \"2cb568ba-79ac-405d-af09-bde6c2b47a53\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7dtn7" Oct 30 07:06:37.042186 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:37.042172 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb568ba-79ac-405d-af09-bde6c2b47a53-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7dtn7\" (UID: \"2cb568ba-79ac-405d-af09-bde6c2b47a53\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7dtn7" Oct 30 07:06:37.042270 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:37.042200 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngd8k\" (UniqueName: \"kubernetes.io/projected/2cb568ba-79ac-405d-af09-bde6c2b47a53-kube-api-access-ngd8k\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7dtn7\" (UID: \"2cb568ba-79ac-405d-af09-bde6c2b47a53\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7dtn7" Oct 30 07:06:37.143135 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:37.143090 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb568ba-79ac-405d-af09-bde6c2b47a53-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7dtn7\" (UID: \"2cb568ba-79ac-405d-af09-bde6c2b47a53\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7dtn7" Oct 30 07:06:37.143346 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:37.143157 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cb568ba-79ac-405d-af09-bde6c2b47a53-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7dtn7\" (UID: \"2cb568ba-79ac-405d-af09-bde6c2b47a53\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7dtn7" Oct 30 07:06:37.143346 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:37.143174 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb568ba-79ac-405d-af09-bde6c2b47a53-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7dtn7\" (UID: \"2cb568ba-79ac-405d-af09-bde6c2b47a53\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7dtn7" Oct 30 07:06:37.143346 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:37.143203 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngd8k\" (UniqueName: \"kubernetes.io/projected/2cb568ba-79ac-405d-af09-bde6c2b47a53-kube-api-access-ngd8k\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7dtn7\" (UID: \"2cb568ba-79ac-405d-af09-bde6c2b47a53\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7dtn7" Oct 30 07:06:37.143346 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:37.143232 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cb568ba-79ac-405d-af09-bde6c2b47a53-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7dtn7\" (UID: \"2cb568ba-79ac-405d-af09-bde6c2b47a53\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7dtn7" Oct 30 07:06:37.143346 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:37.143259 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cb568ba-79ac-405d-af09-bde6c2b47a53-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7dtn7\" (UID: \"2cb568ba-79ac-405d-af09-bde6c2b47a53\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7dtn7" Oct 30 07:06:37.143701 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:37.143675 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb568ba-79ac-405d-af09-bde6c2b47a53-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7dtn7\" (UID: \"2cb568ba-79ac-405d-af09-bde6c2b47a53\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7dtn7" Oct 30 07:06:37.143776 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:37.143717 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb568ba-79ac-405d-af09-bde6c2b47a53-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7dtn7\" (UID: \"2cb568ba-79ac-405d-af09-bde6c2b47a53\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7dtn7" Oct 30 07:06:37.143903 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:37.143884 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cb568ba-79ac-405d-af09-bde6c2b47a53-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7dtn7\" (UID: \"2cb568ba-79ac-405d-af09-bde6c2b47a53\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7dtn7" Oct 30 07:06:37.145683 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:37.145658 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cb568ba-79ac-405d-af09-bde6c2b47a53-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7dtn7\" (UID: \"2cb568ba-79ac-405d-af09-bde6c2b47a53\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7dtn7" Oct 30 07:06:37.145859 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:37.145841 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cb568ba-79ac-405d-af09-bde6c2b47a53-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7dtn7\" (UID: \"2cb568ba-79ac-405d-af09-bde6c2b47a53\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7dtn7" Oct 30 07:06:37.171628 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:37.171595 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngd8k\" (UniqueName: \"kubernetes.io/projected/2cb568ba-79ac-405d-af09-bde6c2b47a53-kube-api-access-ngd8k\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7dtn7\" (UID: \"2cb568ba-79ac-405d-af09-bde6c2b47a53\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7dtn7" Oct 30 07:06:37.258350 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:37.258312 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7dtn7" Oct 30 07:06:37.552867 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:37.552807 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-7dtn7"] Oct 30 07:06:37.555232 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:06:37.555198 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb568ba_79ac_405d_af09_bde6c2b47a53.slice/crio-3f3238d71f73d18ac67d85f11c973b7e6da7469d9943f8be9f295ad6b8e18913 WatchSource:0}: Error finding container 3f3238d71f73d18ac67d85f11c973b7e6da7469d9943f8be9f295ad6b8e18913: Status 404 returned error can't find the container with id 3f3238d71f73d18ac67d85f11c973b7e6da7469d9943f8be9f295ad6b8e18913 Oct 30 07:06:38.232239 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:38.232137 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7dtn7" event={"ID":"2cb568ba-79ac-405d-af09-bde6c2b47a53","Type":"ContainerStarted","Data":"5a5f494f0354da4a689e7361e70f32eba84e098f8e61b88708ebf66e67fe162f"} Oct 30 07:06:38.232239 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:38.232196 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7dtn7" event={"ID":"2cb568ba-79ac-405d-af09-bde6c2b47a53","Type":"ContainerStarted","Data":"3f3238d71f73d18ac67d85f11c973b7e6da7469d9943f8be9f295ad6b8e18913"} Oct 30 07:06:38.259037 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:38.259011 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7dtn7" Oct 30 07:06:38.275431 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:38.275385 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7dtn7" podStartSLOduration=2.078589792 podStartE2EDuration="2.275372031s" totalImagesPullingTime="196.782239ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:06:38.274303006 +0000 UTC m=+604.617350995" watchObservedRunningTime="2026-10-30 07:06:38.275372031 +0000 UTC m=+604.618419996" Oct 30 07:06:41.281403 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:41.281368 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-xnj7x"] Oct 30 07:06:41.285611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:41.285590 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-xnj7x" Oct 30 07:06:41.299744 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:41.299713 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 30 07:06:41.319559 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:41.319522 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-xnj7x"] Oct 30 07:06:41.381745 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:41.381710 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqgmg\" (UniqueName: \"kubernetes.io/projected/4ec98545-1b89-4b01-883f-098acce664e7-kube-api-access-kqgmg\") pod \"echo-simulator-kserve-64b6f8f96c-xnj7x\" (UID: \"4ec98545-1b89-4b01-883f-098acce664e7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xnj7x" Oct 30 07:06:41.381892 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:41.381761 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ec98545-1b89-4b01-883f-098acce664e7-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-xnj7x\" (UID: \"4ec98545-1b89-4b01-883f-098acce664e7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xnj7x" Oct 30 07:06:41.381892 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:41.381854 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ec98545-1b89-4b01-883f-098acce664e7-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-xnj7x\" (UID: \"4ec98545-1b89-4b01-883f-098acce664e7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xnj7x" Oct 30 07:06:41.381993 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:41.381901 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ec98545-1b89-4b01-883f-098acce664e7-home\") pod \"echo-simulator-kserve-64b6f8f96c-xnj7x\" (UID: \"4ec98545-1b89-4b01-883f-098acce664e7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xnj7x" Oct 30 07:06:41.381993 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:41.381930 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ec98545-1b89-4b01-883f-098acce664e7-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-xnj7x\" (UID: \"4ec98545-1b89-4b01-883f-098acce664e7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xnj7x" Oct 30 07:06:41.381993 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:41.381947 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ec98545-1b89-4b01-883f-098acce664e7-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-xnj7x\" (UID: \"4ec98545-1b89-4b01-883f-098acce664e7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xnj7x" Oct 30 07:06:41.482976 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:41.482936 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ec98545-1b89-4b01-883f-098acce664e7-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-xnj7x\" (UID: \"4ec98545-1b89-4b01-883f-098acce664e7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xnj7x" Oct 30 07:06:41.482976 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:41.482981 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ec98545-1b89-4b01-883f-098acce664e7-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-xnj7x\" (UID: \"4ec98545-1b89-4b01-883f-098acce664e7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xnj7x" Oct 30 07:06:41.483237 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:41.483000 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ec98545-1b89-4b01-883f-098acce664e7-home\") pod \"echo-simulator-kserve-64b6f8f96c-xnj7x\" (UID: \"4ec98545-1b89-4b01-883f-098acce664e7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xnj7x" Oct 30 07:06:41.483237 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:41.483026 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ec98545-1b89-4b01-883f-098acce664e7-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-xnj7x\" (UID: \"4ec98545-1b89-4b01-883f-098acce664e7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xnj7x" Oct 30 07:06:41.483237 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:41.483055 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ec98545-1b89-4b01-883f-098acce664e7-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-xnj7x\" (UID: \"4ec98545-1b89-4b01-883f-098acce664e7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xnj7x" Oct 30 07:06:41.483237 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:41.483090 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqgmg\" (UniqueName: \"kubernetes.io/projected/4ec98545-1b89-4b01-883f-098acce664e7-kube-api-access-kqgmg\") pod \"echo-simulator-kserve-64b6f8f96c-xnj7x\" (UID: \"4ec98545-1b89-4b01-883f-098acce664e7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xnj7x" Oct 30 07:06:41.483550 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:41.483512 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ec98545-1b89-4b01-883f-098acce664e7-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-xnj7x\" (UID: \"4ec98545-1b89-4b01-883f-098acce664e7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xnj7x" Oct 30 07:06:41.483550 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:41.483534 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ec98545-1b89-4b01-883f-098acce664e7-home\") pod \"echo-simulator-kserve-64b6f8f96c-xnj7x\" (UID: \"4ec98545-1b89-4b01-883f-098acce664e7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xnj7x" Oct 30 07:06:41.483730 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:41.483566 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ec98545-1b89-4b01-883f-098acce664e7-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-xnj7x\" (UID: \"4ec98545-1b89-4b01-883f-098acce664e7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xnj7x" Oct 30 07:06:41.485517 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:41.485490 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ec98545-1b89-4b01-883f-098acce664e7-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-xnj7x\" (UID: \"4ec98545-1b89-4b01-883f-098acce664e7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xnj7x" Oct 30 07:06:41.485943 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:41.485923 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ec98545-1b89-4b01-883f-098acce664e7-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-xnj7x\" (UID: \"4ec98545-1b89-4b01-883f-098acce664e7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xnj7x" Oct 30 07:06:41.507221 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:41.507199 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqgmg\" (UniqueName: \"kubernetes.io/projected/4ec98545-1b89-4b01-883f-098acce664e7-kube-api-access-kqgmg\") pod \"echo-simulator-kserve-64b6f8f96c-xnj7x\" (UID: \"4ec98545-1b89-4b01-883f-098acce664e7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xnj7x" Oct 30 07:06:41.600631 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:41.600518 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-xnj7x" Oct 30 07:06:41.907759 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:41.907731 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-xnj7x"] Oct 30 07:06:41.909988 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:06:41.909954 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ec98545_1b89_4b01_883f_098acce664e7.slice/crio-d8e6c6dddb30b883fb2722c311bd0b5c46e7bbec0383897d5299a3b769ec6fe0 WatchSource:0}: Error finding container d8e6c6dddb30b883fb2722c311bd0b5c46e7bbec0383897d5299a3b769ec6fe0: Status 404 returned error can't find the container with id d8e6c6dddb30b883fb2722c311bd0b5c46e7bbec0383897d5299a3b769ec6fe0 Oct 30 07:06:42.251557 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:42.251524 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-xnj7x" event={"ID":"4ec98545-1b89-4b01-883f-098acce664e7","Type":"ContainerStarted","Data":"d8e6c6dddb30b883fb2722c311bd0b5c46e7bbec0383897d5299a3b769ec6fe0"} Oct 30 07:06:42.928256 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:42.928221 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k"] Oct 30 07:06:42.932330 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:42.932298 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k" Oct 30 07:06:42.957730 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:42.957697 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 07:06:42.959548 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:42.959345 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k"] Oct 30 07:06:43.098542 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:43.098504 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75t6x\" (UniqueName: \"kubernetes.io/projected/2a4b5a5d-94d1-47d0-9bbe-3849812d8ece-kube-api-access-75t6x\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k\" (UID: \"2a4b5a5d-94d1-47d0-9bbe-3849812d8ece\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k" Oct 30 07:06:43.098755 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:43.098556 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a4b5a5d-94d1-47d0-9bbe-3849812d8ece-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k\" (UID: \"2a4b5a5d-94d1-47d0-9bbe-3849812d8ece\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k" Oct 30 07:06:43.098755 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:43.098624 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a4b5a5d-94d1-47d0-9bbe-3849812d8ece-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k\" (UID: \"2a4b5a5d-94d1-47d0-9bbe-3849812d8ece\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k" Oct 30 07:06:43.098755 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:43.098681 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a4b5a5d-94d1-47d0-9bbe-3849812d8ece-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k\" (UID: \"2a4b5a5d-94d1-47d0-9bbe-3849812d8ece\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k" Oct 30 07:06:43.098885 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:43.098769 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a4b5a5d-94d1-47d0-9bbe-3849812d8ece-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k\" (UID: \"2a4b5a5d-94d1-47d0-9bbe-3849812d8ece\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k" Oct 30 07:06:43.098885 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:43.098796 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a4b5a5d-94d1-47d0-9bbe-3849812d8ece-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k\" (UID: \"2a4b5a5d-94d1-47d0-9bbe-3849812d8ece\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k" Oct 30 07:06:43.199326 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:43.199230 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a4b5a5d-94d1-47d0-9bbe-3849812d8ece-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k\" (UID: \"2a4b5a5d-94d1-47d0-9bbe-3849812d8ece\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k" Oct 30 07:06:43.199326 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:43.199282 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a4b5a5d-94d1-47d0-9bbe-3849812d8ece-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k\" (UID: \"2a4b5a5d-94d1-47d0-9bbe-3849812d8ece\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k" Oct 30 07:06:43.199557 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:43.199342 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75t6x\" (UniqueName: \"kubernetes.io/projected/2a4b5a5d-94d1-47d0-9bbe-3849812d8ece-kube-api-access-75t6x\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k\" (UID: \"2a4b5a5d-94d1-47d0-9bbe-3849812d8ece\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k" Oct 30 07:06:43.199557 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:43.199382 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a4b5a5d-94d1-47d0-9bbe-3849812d8ece-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k\" (UID: \"2a4b5a5d-94d1-47d0-9bbe-3849812d8ece\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k" Oct 30 07:06:43.199557 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:43.199406 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a4b5a5d-94d1-47d0-9bbe-3849812d8ece-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k\" (UID: \"2a4b5a5d-94d1-47d0-9bbe-3849812d8ece\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k" Oct 30 07:06:43.199557 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:43.199454 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a4b5a5d-94d1-47d0-9bbe-3849812d8ece-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k\" (UID: \"2a4b5a5d-94d1-47d0-9bbe-3849812d8ece\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k" Oct 30 07:06:43.199816 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:43.199786 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a4b5a5d-94d1-47d0-9bbe-3849812d8ece-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k\" (UID: \"2a4b5a5d-94d1-47d0-9bbe-3849812d8ece\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k" Oct 30 07:06:43.199873 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:43.199848 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a4b5a5d-94d1-47d0-9bbe-3849812d8ece-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k\" (UID: \"2a4b5a5d-94d1-47d0-9bbe-3849812d8ece\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k" Oct 30 07:06:43.199919 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:43.199874 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a4b5a5d-94d1-47d0-9bbe-3849812d8ece-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k\" (UID: \"2a4b5a5d-94d1-47d0-9bbe-3849812d8ece\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k" Oct 30 07:06:43.201919 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:43.201890 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a4b5a5d-94d1-47d0-9bbe-3849812d8ece-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k\" (UID: \"2a4b5a5d-94d1-47d0-9bbe-3849812d8ece\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k" Oct 30 07:06:43.202287 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:43.202265 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a4b5a5d-94d1-47d0-9bbe-3849812d8ece-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k\" (UID: \"2a4b5a5d-94d1-47d0-9bbe-3849812d8ece\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k" Oct 30 07:06:43.202501 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:43.202481 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8ksd" Oct 30 07:06:43.220015 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:43.219984 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8ksd" Oct 30 07:06:43.224015 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:43.223987 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75t6x\" (UniqueName: \"kubernetes.io/projected/2a4b5a5d-94d1-47d0-9bbe-3849812d8ece-kube-api-access-75t6x\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k\" (UID: \"2a4b5a5d-94d1-47d0-9bbe-3849812d8ece\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k" Oct 30 07:06:43.249776 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:43.249738 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k" Oct 30 07:06:43.270649 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:43.270617 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8ksd" Oct 30 07:06:43.555011 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:43.554629 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k"] Oct 30 07:06:43.556791 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:06:43.556765 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4b5a5d_94d1_47d0_9bbe_3849812d8ece.slice/crio-c4cee6ba04952243e2ea3336be37db1b8761eece2c2557a603143cbb49547ffb WatchSource:0}: Error finding container c4cee6ba04952243e2ea3336be37db1b8761eece2c2557a603143cbb49547ffb: Status 404 returned error can't find the container with id c4cee6ba04952243e2ea3336be37db1b8761eece2c2557a603143cbb49547ffb Oct 30 07:06:44.263532 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:44.263492 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k" event={"ID":"2a4b5a5d-94d1-47d0-9bbe-3849812d8ece","Type":"ContainerStarted","Data":"f5486bef61675a6194e2445adf1a87274db881e87581451d93d2149aa20040c4"} Oct 30 07:06:44.264062 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:44.263555 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k" event={"ID":"2a4b5a5d-94d1-47d0-9bbe-3849812d8ece","Type":"ContainerStarted","Data":"c4cee6ba04952243e2ea3336be37db1b8761eece2c2557a603143cbb49547ffb"} Oct 30 07:06:44.265197 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:44.265165 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-xnj7x" event={"ID":"4ec98545-1b89-4b01-883f-098acce664e7","Type":"ContainerStarted","Data":"dc4ab97c5ae8579cb0907705a86841cda860b40114abddd4f3909ed71359698a"} Oct 30 07:06:44.305309 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:44.305255 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k" podStartSLOduration=2.129363088 podStartE2EDuration="2.305239995s" totalImagesPullingTime="175.876907ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:06:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:06:44.304720326 +0000 UTC m=+610.647768291" watchObservedRunningTime="2026-10-30 07:06:44.305239995 +0000 UTC m=+610.648287959" Oct 30 07:06:44.348421 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:44.348359 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-xnj7x" podStartSLOduration=1.7219162940000001 podStartE2EDuration="3.348340043s" totalImagesPullingTime="1.626423749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:06:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:06:44.347610053 +0000 UTC m=+610.690658017" watchObservedRunningTime="2026-10-30 07:06:44.348340043 +0000 UTC m=+610.691388009" Oct 30 07:06:44.601245 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:44.601165 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-xnj7x" Oct 30 07:06:45.250587 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:45.250547 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k" Oct 30 07:06:47.259200 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:47.259168 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7dtn7" Oct 30 07:06:47.276476 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:47.276448 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7dtn7" Oct 30 07:06:47.291315 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:47.291281 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7dtn7" Oct 30 07:06:51.601287 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:51.601251 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-xnj7x" Oct 30 07:06:51.614157 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:51.614132 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-xnj7x" Oct 30 07:06:52.314253 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:52.314221 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-xnj7x" Oct 30 07:06:53.021597 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:06:53.021544 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:06:53.250750 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:53.250713 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k" Oct 30 07:06:53.263411 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:53.263381 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k" Oct 30 07:06:53.322624 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:53.322552 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k" Oct 30 07:06:55.716178 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:55.716144 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82"] Oct 30 07:06:55.719862 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:55.719842 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:06:55.748483 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:55.748446 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-j9rfz\"" Oct 30 07:06:55.748728 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:55.748677 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82"] Oct 30 07:06:55.812986 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:55.812956 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-4bj82\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:06:55.812986 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:55.812988 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9328c8af-90ee-4949-be97-f3e6c04416a2-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-4bj82\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:06:55.813228 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:55.813019 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-4bj82\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:06:55.813228 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:55.813073 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9328c8af-90ee-4949-be97-f3e6c04416a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-4bj82\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:06:55.813228 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:55.813175 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxcr6\" (UniqueName: \"kubernetes.io/projected/9328c8af-90ee-4949-be97-f3e6c04416a2-kube-api-access-mxcr6\") pod \"maas-default-gateway-openshift-default-6dcf757899-4bj82\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:06:55.813228 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:55.813214 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-4bj82\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:06:55.813418 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:55.813236 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-4bj82\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:06:55.813418 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:55.813267 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9328c8af-90ee-4949-be97-f3e6c04416a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-4bj82\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:06:55.813418 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:55.813353 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-4bj82\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:06:55.914441 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:55.914390 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-4bj82\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:06:55.914441 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:55.914443 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-4bj82\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:06:55.914702 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:55.914473 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9328c8af-90ee-4949-be97-f3e6c04416a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-4bj82\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:06:55.914702 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:55.914504 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-4bj82\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:06:55.914702 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:55.914554 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-4bj82\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:06:55.914702 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:55.914595 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9328c8af-90ee-4949-be97-f3e6c04416a2-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-4bj82\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:06:55.914702 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:55.914637 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-4bj82\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:06:55.914702 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:55.914664 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9328c8af-90ee-4949-be97-f3e6c04416a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-4bj82\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:06:55.914702 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:55.914696 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxcr6\" (UniqueName: \"kubernetes.io/projected/9328c8af-90ee-4949-be97-f3e6c04416a2-kube-api-access-mxcr6\") pod \"maas-default-gateway-openshift-default-6dcf757899-4bj82\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:06:55.915016 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:55.914860 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-4bj82\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:06:55.915016 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:55.914927 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-4bj82\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:06:55.915222 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:55.915195 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-4bj82\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:06:55.915291 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:55.915245 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-4bj82\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:06:55.915506 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:55.915486 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9328c8af-90ee-4949-be97-f3e6c04416a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-4bj82\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:06:55.916824 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:55.916801 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-4bj82\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:06:55.917042 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:55.917025 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9328c8af-90ee-4949-be97-f3e6c04416a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-4bj82\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:06:55.936793 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:55.936765 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9328c8af-90ee-4949-be97-f3e6c04416a2-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-4bj82\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:06:55.936888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:55.936862 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxcr6\" (UniqueName: \"kubernetes.io/projected/9328c8af-90ee-4949-be97-f3e6c04416a2-kube-api-access-mxcr6\") pod \"maas-default-gateway-openshift-default-6dcf757899-4bj82\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:06:56.032982 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:56.032947 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:06:56.350232 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:56.350170 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82"] Oct 30 07:06:56.352320 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:06:56.352293 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9328c8af_90ee_4949_be97_f3e6c04416a2.slice/crio-f18ca9b4c51be903f14af32b6d6c16916519d33e10d52b9d83509901bcee6d37 WatchSource:0}: Error finding container f18ca9b4c51be903f14af32b6d6c16916519d33e10d52b9d83509901bcee6d37: Status 404 returned error can't find the container with id f18ca9b4c51be903f14af32b6d6c16916519d33e10d52b9d83509901bcee6d37 Oct 30 07:06:57.325861 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:06:57.325822 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" event={"ID":"9328c8af-90ee-4949-be97-f3e6c04416a2","Type":"ContainerStarted","Data":"f18ca9b4c51be903f14af32b6d6c16916519d33e10d52b9d83509901bcee6d37"} Oct 30 07:07:10.752275 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.752238 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx"] Oct 30 07:07:10.760587 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.760558 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:07:10.791047 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.791024 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx"] Oct 30 07:07:10.799346 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.799321 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd"] Oct 30 07:07:10.802968 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.802948 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:07:10.830864 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.830834 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd"] Oct 30 07:07:10.836832 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.836793 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-d2cqx\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:07:10.837082 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.837058 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-thknd\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:07:10.837206 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.837127 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-thknd\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:07:10.837206 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.837158 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-thknd\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:07:10.837206 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.837185 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-thknd\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:07:10.837206 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.837200 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jtmn\" (UniqueName: \"kubernetes.io/projected/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-kube-api-access-4jtmn\") pod \"maas-default-gateway-openshift-default-6dcf757899-thknd\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:07:10.837406 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.837218 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-thknd\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:07:10.837406 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.837251 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a320a4da-997e-4463-bdd2-6792b153bf72-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-d2cqx\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:07:10.837406 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.837292 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-d2cqx\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:07:10.837406 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.837306 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a320a4da-997e-4463-bdd2-6792b153bf72-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-d2cqx\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:07:10.837406 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.837326 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdw25\" (UniqueName: \"kubernetes.io/projected/a320a4da-997e-4463-bdd2-6792b153bf72-kube-api-access-fdw25\") pod \"maas-default-gateway-openshift-default-6dcf757899-d2cqx\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:07:10.837406 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.837365 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-thknd\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:07:10.837733 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.837430 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-d2cqx\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:07:10.837733 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.837449 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-thknd\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:07:10.837733 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.837463 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-d2cqx\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:07:10.837733 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.837489 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-d2cqx\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:07:10.837733 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.837525 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-thknd\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:07:10.837733 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.837541 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a320a4da-997e-4463-bdd2-6792b153bf72-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-d2cqx\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:07:10.938779 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.938739 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-d2cqx\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:07:10.938779 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.938775 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-thknd\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:07:10.938779 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.938791 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-d2cqx\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:07:10.939090 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.938810 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-d2cqx\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:07:10.939090 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.938838 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-thknd\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:07:10.939090 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.938881 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a320a4da-997e-4463-bdd2-6792b153bf72-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-d2cqx\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:07:10.939297 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.939271 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-thknd\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:07:10.939375 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.939301 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-d2cqx\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:07:10.939375 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.939301 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-thknd\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:07:10.939375 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.939340 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-d2cqx\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:07:10.939375 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.939276 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-d2cqx\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:07:10.939591 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.939414 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-thknd\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:07:10.939591 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.939454 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-thknd\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:07:10.939591 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.939480 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-thknd\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:07:10.939591 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.939509 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-thknd\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:07:10.939591 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.939532 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jtmn\" (UniqueName: \"kubernetes.io/projected/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-kube-api-access-4jtmn\") pod \"maas-default-gateway-openshift-default-6dcf757899-thknd\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:07:10.939591 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.939565 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-thknd\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:07:10.939936 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.939604 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a320a4da-997e-4463-bdd2-6792b153bf72-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-d2cqx\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:07:10.939936 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.939630 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-d2cqx\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:07:10.939936 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.939649 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-d2cqx\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:07:10.939936 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.939821 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-thknd\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:07:10.940243 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.939939 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-d2cqx\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:07:10.940243 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.940005 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a320a4da-997e-4463-bdd2-6792b153bf72-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-d2cqx\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:07:10.940243 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.940071 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdw25\" (UniqueName: \"kubernetes.io/projected/a320a4da-997e-4463-bdd2-6792b153bf72-kube-api-access-fdw25\") pod \"maas-default-gateway-openshift-default-6dcf757899-d2cqx\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:07:10.940243 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.940132 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-thknd\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:07:10.940508 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.940461 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-thknd\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:07:10.940508 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.940472 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-thknd\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:07:10.941155 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.941125 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a320a4da-997e-4463-bdd2-6792b153bf72-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-d2cqx\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:07:10.941707 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.941680 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-d2cqx\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:07:10.942393 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.942372 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-thknd\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:07:10.943040 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.943019 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-thknd\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:07:10.943228 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.943206 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a320a4da-997e-4463-bdd2-6792b153bf72-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-d2cqx\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:07:10.963203 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.963162 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jtmn\" (UniqueName: \"kubernetes.io/projected/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-kube-api-access-4jtmn\") pod \"maas-default-gateway-openshift-default-6dcf757899-thknd\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:07:10.963386 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.963318 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a320a4da-997e-4463-bdd2-6792b153bf72-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-d2cqx\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:07:10.963386 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.963329 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-thknd\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:07:10.963524 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:10.963501 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdw25\" (UniqueName: \"kubernetes.io/projected/a320a4da-997e-4463-bdd2-6792b153bf72-kube-api-access-fdw25\") pod \"maas-default-gateway-openshift-default-6dcf757899-d2cqx\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:07:11.074770 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:11.074681 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:07:11.118303 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:11.118264 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:07:11.381544 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:11.381511 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx"] Oct 30 07:07:11.383327 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:07:11.383303 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda320a4da_997e_4463_bdd2_6792b153bf72.slice/crio-5e95a5eeb2c8b576faf78a9f65ad7c2829ab6367e21e47f436dcd8262cc630c8 WatchSource:0}: Error finding container 5e95a5eeb2c8b576faf78a9f65ad7c2829ab6367e21e47f436dcd8262cc630c8: Status 404 returned error can't find the container with id 5e95a5eeb2c8b576faf78a9f65ad7c2829ab6367e21e47f436dcd8262cc630c8 Oct 30 07:07:11.395181 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:11.395151 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" event={"ID":"a320a4da-997e-4463-bdd2-6792b153bf72","Type":"ContainerStarted","Data":"5e95a5eeb2c8b576faf78a9f65ad7c2829ab6367e21e47f436dcd8262cc630c8"} Oct 30 07:07:11.432174 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:11.432147 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd"] Oct 30 07:07:11.434469 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:07:11.434446 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33872c4d_c6f0_4f12_9daf_4c23cb7b7d72.slice/crio-03ebe115f09ead8e434920cbc974d5c68ed12d8da2c5f605b64de6ee96366522 WatchSource:0}: Error finding container 03ebe115f09ead8e434920cbc974d5c68ed12d8da2c5f605b64de6ee96366522: Status 404 returned error can't find the container with id 03ebe115f09ead8e434920cbc974d5c68ed12d8da2c5f605b64de6ee96366522 Oct 30 07:07:12.403550 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:12.403510 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" event={"ID":"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72","Type":"ContainerStarted","Data":"03ebe115f09ead8e434920cbc974d5c68ed12d8da2c5f605b64de6ee96366522"} Oct 30 07:07:14.716415 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:14.716369 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:07:14.716831 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:14.716473 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:07:14.716831 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:14.716519 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:07:14.721855 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:14.721814 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:07:14.721957 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:14.721898 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:07:14.721957 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:14.721943 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:07:15.047723 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:15.047671 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:07:15.047839 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:15.047754 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:07:15.047839 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:15.047794 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:07:15.429651 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:15.425839 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" event={"ID":"9328c8af-90ee-4949-be97-f3e6c04416a2","Type":"ContainerStarted","Data":"ca0668842600a7ce99755aa25e95e315540eb32a426b75ab9aabfc795bee82a5"} Oct 30 07:07:15.479601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:15.478997 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" event={"ID":"a320a4da-997e-4463-bdd2-6792b153bf72","Type":"ContainerStarted","Data":"a7b632e9f2e8bf195748fe3b74654832c63b3ed7a4fd5a07353cea024302093f"} Oct 30 07:07:15.483858 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:15.481995 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" event={"ID":"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72","Type":"ContainerStarted","Data":"4c1c6de51f28ae2ae1f910f5b000f171e2e7a3baa1506d0564854093cd38cb24"} Oct 30 07:07:15.500374 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:15.500313 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" podStartSLOduration=1.807591854 podStartE2EDuration="20.500298876s" totalImagesPullingTime="18.692707022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:06:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:07:15.488509322 +0000 UTC m=+641.831557313" watchObservedRunningTime="2026-10-30 07:07:15.500298876 +0000 UTC m=+641.843346919" Oct 30 07:07:15.585246 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:15.585058 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" podStartSLOduration=2.300367337 podStartE2EDuration="5.585033801s" totalImagesPullingTime="3.284666464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:07:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:07:15.584797718 +0000 UTC m=+641.927845701" watchObservedRunningTime="2026-10-30 07:07:15.585033801 +0000 UTC m=+641.928081760" Oct 30 07:07:15.586275 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:15.586227 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" podStartSLOduration=2.2559736 podStartE2EDuration="5.586219178s" totalImagesPullingTime="3.330245578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:07:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:07:15.533882202 +0000 UTC m=+641.876930169" watchObservedRunningTime="2026-10-30 07:07:15.586219178 +0000 UTC m=+641.929267149" Oct 30 07:07:16.036851 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:16.036813 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:07:16.037892 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:16.037536 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-4bj82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Oct 30 07:07:16.037892 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:16.037611 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" podUID="9328c8af-90ee-4949-be97-f3e6c04416a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Oct 30 07:07:16.078597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:16.078380 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:07:16.091599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:16.087716 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-d2cqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 30 07:07:16.091599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:16.087783 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" podUID="a320a4da-997e-4463-bdd2-6792b153bf72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 30 07:07:16.118417 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:16.118381 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:07:16.119909 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:16.119886 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-thknd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 30 07:07:16.120024 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:16.119938 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" podUID="33872c4d-c6f0-4f12-9daf-4c23cb7b7d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 30 07:07:17.038596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:17.038051 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-4bj82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Oct 30 07:07:17.038596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:17.038117 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" podUID="9328c8af-90ee-4949-be97-f3e6c04416a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Oct 30 07:07:17.081599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:17.081373 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-d2cqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 30 07:07:17.081599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:17.081465 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" podUID="a320a4da-997e-4463-bdd2-6792b153bf72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 30 07:07:17.119148 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:17.119104 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-thknd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 30 07:07:17.119327 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:17.119163 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" podUID="33872c4d-c6f0-4f12-9daf-4c23cb7b7d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 30 07:07:18.046605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:18.042992 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-4bj82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Oct 30 07:07:18.046605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:18.043056 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" podUID="9328c8af-90ee-4949-be97-f3e6c04416a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Oct 30 07:07:18.077598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:18.077077 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-d2cqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 30 07:07:18.077598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:18.077171 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" podUID="a320a4da-997e-4463-bdd2-6792b153bf72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 30 07:07:18.119071 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:18.119030 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-thknd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 30 07:07:18.119258 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:18.119085 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" podUID="33872c4d-c6f0-4f12-9daf-4c23cb7b7d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 30 07:07:19.037597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:19.034953 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-4bj82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Oct 30 07:07:19.037597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:19.035012 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" podUID="9328c8af-90ee-4949-be97-f3e6c04416a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Oct 30 07:07:19.079594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:19.079317 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-d2cqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 30 07:07:19.079594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:19.079394 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" podUID="a320a4da-997e-4463-bdd2-6792b153bf72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 30 07:07:19.118811 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:19.118772 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-thknd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 30 07:07:19.118979 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:19.118824 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" podUID="33872c4d-c6f0-4f12-9daf-4c23cb7b7d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 30 07:07:20.041007 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:20.040964 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-4bj82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Oct 30 07:07:20.041323 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:20.041291 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" podUID="9328c8af-90ee-4949-be97-f3e6c04416a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Oct 30 07:07:20.080600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:20.079277 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-d2cqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 30 07:07:20.080600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:20.079358 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" podUID="a320a4da-997e-4463-bdd2-6792b153bf72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 30 07:07:20.119400 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:20.119361 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-thknd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 30 07:07:20.119622 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:20.119424 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" podUID="33872c4d-c6f0-4f12-9daf-4c23cb7b7d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 30 07:07:21.034054 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:21.034010 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-4bj82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Oct 30 07:07:21.034375 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:21.034339 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" podUID="9328c8af-90ee-4949-be97-f3e6c04416a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Oct 30 07:07:21.075403 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:21.075369 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-d2cqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 30 07:07:21.075590 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:21.075426 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" podUID="a320a4da-997e-4463-bdd2-6792b153bf72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 30 07:07:21.119300 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:21.119253 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-thknd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 30 07:07:21.119763 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:21.119327 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" podUID="33872c4d-c6f0-4f12-9daf-4c23cb7b7d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 30 07:07:22.034310 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:22.034269 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-4bj82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Oct 30 07:07:22.034496 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:22.034338 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" podUID="9328c8af-90ee-4949-be97-f3e6c04416a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Oct 30 07:07:22.076033 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:22.075991 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-d2cqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 30 07:07:22.076211 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:22.076067 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" podUID="a320a4da-997e-4463-bdd2-6792b153bf72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 30 07:07:22.119485 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:22.119439 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-thknd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 30 07:07:22.119902 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:22.119510 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" podUID="33872c4d-c6f0-4f12-9daf-4c23cb7b7d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 30 07:07:23.036455 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:07:23.036377 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:07:23.521964 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:23.521925 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:07:23.522742 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:23.522713 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:07:23.523852 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:23.523829 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:07:23.675964 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:23.675932 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:07:23.676820 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:23.676803 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:07:23.677561 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:23.677540 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:07:23.899120 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:23.899036 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:07:23.899838 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:23.899812 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:07:23.900753 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:23.900731 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:07:43.295423 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.295388 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq"] Oct 30 07:07:43.300287 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.300260 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:07:43.315430 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.315408 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9bd1c95c-openshift-default-dockercfg-66zzs\"" Oct 30 07:07:43.331333 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.331306 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq"] Oct 30 07:07:43.366274 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.366238 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-workload-certs\") pod \"e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:07:43.366410 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.366290 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istiod-ca-cert\") pod \"e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:07:43.366466 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.366399 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kztg5\" (UniqueName: \"kubernetes.io/projected/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-kube-api-access-kztg5\") pod \"e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:07:43.366466 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.366456 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istio-podinfo\") pod \"e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:07:43.366544 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.366499 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istio-data\") pod \"e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:07:43.366544 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.366528 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-workload-socket\") pod \"e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:07:43.366654 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.366627 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-credential-socket\") pod \"e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:07:43.366718 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.366702 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istio-token\") pod \"e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:07:43.366775 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.366752 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istio-envoy\") pod \"e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:07:43.442168 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.442132 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr"] Oct 30 07:07:43.445800 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.445777 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:07:43.467273 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.467249 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kztg5\" (UniqueName: \"kubernetes.io/projected/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-kube-api-access-kztg5\") pod \"e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:07:43.467420 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.467304 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istio-podinfo\") pod \"e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:07:43.467420 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.467334 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istio-data\") pod \"e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:07:43.467420 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.467373 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-workload-socket\") pod \"e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:07:43.467420 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.467400 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-credential-socket\") pod \"e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:07:43.467647 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.467458 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istio-token\") pod \"e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:07:43.467647 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.467491 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istio-envoy\") pod \"e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:07:43.467647 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.467536 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-workload-certs\") pod \"e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:07:43.468011 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.467941 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-credential-socket\") pod \"e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:07:43.468011 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.467953 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-7a77072b-openshift-default-dockercfg-8xnxf\"" Oct 30 07:07:43.468188 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.468040 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-workload-socket\") pod \"e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:07:43.468188 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.468044 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istiod-ca-cert\") pod \"e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:07:43.468188 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.468097 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istio-data\") pod \"e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:07:43.468353 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.468271 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-workload-certs\") pod \"e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:07:43.468605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.468583 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istiod-ca-cert\") pod \"e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:07:43.470180 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.470159 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istio-envoy\") pod \"e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:07:43.471763 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.471743 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istio-podinfo\") pod \"e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:07:43.473353 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.473329 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr"] Oct 30 07:07:43.495120 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.495091 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kztg5\" (UniqueName: \"kubernetes.io/projected/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-kube-api-access-kztg5\") pod \"e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:07:43.502969 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.502944 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istio-token\") pod \"e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:07:43.568754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.568677 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-credential-socket\") pod \"e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:07:43.568911 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.568782 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-istio-data\") pod \"e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:07:43.568911 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.568867 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2895d955-4a1f-45f1-94cb-251e4996c772-istiod-ca-cert\") pod \"e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:07:43.569010 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.568935 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2895d955-4a1f-45f1-94cb-251e4996c772-istio-token\") pod \"e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:07:43.569010 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.568973 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-workload-certs\") pod \"e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:07:43.569010 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.569003 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-workload-socket\") pod \"e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:07:43.569171 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.569080 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-627tq\" (UniqueName: \"kubernetes.io/projected/2895d955-4a1f-45f1-94cb-251e4996c772-kube-api-access-627tq\") pod \"e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:07:43.569171 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.569105 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-istio-envoy\") pod \"e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:07:43.569276 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.569206 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2895d955-4a1f-45f1-94cb-251e4996c772-istio-podinfo\") pod \"e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:07:43.615635 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.615599 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:07:43.670295 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.670254 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2895d955-4a1f-45f1-94cb-251e4996c772-istiod-ca-cert\") pod \"e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:07:43.670498 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.670304 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2895d955-4a1f-45f1-94cb-251e4996c772-istio-token\") pod \"e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:07:43.670498 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.670339 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-workload-certs\") pod \"e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:07:43.670498 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.670374 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-workload-socket\") pod \"e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:07:43.670498 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.670414 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-627tq\" (UniqueName: \"kubernetes.io/projected/2895d955-4a1f-45f1-94cb-251e4996c772-kube-api-access-627tq\") pod \"e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:07:43.670498 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.670436 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-istio-envoy\") pod \"e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:07:43.670791 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.670504 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2895d955-4a1f-45f1-94cb-251e4996c772-istio-podinfo\") pod \"e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:07:43.670791 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.670607 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-credential-socket\") pod \"e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:07:43.670791 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.670641 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-istio-data\") pod \"e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:07:43.671146 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.671120 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-istio-data\") pod \"e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:07:43.671240 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.671183 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-workload-socket\") pod \"e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:07:43.671317 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.671294 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-workload-certs\") pod \"e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:07:43.671601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.671522 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-credential-socket\") pod \"e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:07:43.672040 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.672015 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2895d955-4a1f-45f1-94cb-251e4996c772-istiod-ca-cert\") pod \"e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:07:43.673455 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.673430 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-istio-envoy\") pod \"e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:07:43.673551 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.673452 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2895d955-4a1f-45f1-94cb-251e4996c772-istio-podinfo\") pod \"e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:07:43.719680 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.711063 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2895d955-4a1f-45f1-94cb-251e4996c772-istio-token\") pod \"e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:07:43.719680 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.718341 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-627tq\" (UniqueName: \"kubernetes.io/projected/2895d955-4a1f-45f1-94cb-251e4996c772-kube-api-access-627tq\") pod \"e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:07:43.760405 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.760319 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:07:43.895725 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.895654 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx"] Oct 30 07:07:43.900509 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.900487 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:07:43.916225 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.916204 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-a5d88019-gw-openshift-default-dockercfg-llq5x\"" Oct 30 07:07:43.925224 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.925191 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx"] Oct 30 07:07:43.946918 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.946892 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq"] Oct 30 07:07:43.948850 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:07:43.948824 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e2d9f9d_3fe2_4abe_8136_f34cdf8de323.slice/crio-b5cc6811ef408ff64fd9f6a630c2af2f0e40b2a2269e6c58178a86f48e90e724 WatchSource:0}: Error finding container b5cc6811ef408ff64fd9f6a630c2af2f0e40b2a2269e6c58178a86f48e90e724: Status 404 returned error can't find the container with id b5cc6811ef408ff64fd9f6a630c2af2f0e40b2a2269e6c58178a86f48e90e724 Oct 30 07:07:43.952278 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.952246 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:07:43.952345 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.952310 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:07:43.952345 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.952338 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:07:43.973467 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.973445 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istio-envoy\") pod \"e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:07:43.973563 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.973491 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-credential-socket\") pod \"e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:07:43.973563 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.973514 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istio-data\") pod \"e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:07:43.973677 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.973657 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istio-podinfo\") pod \"e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:07:43.973716 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.973685 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istiod-ca-cert\") pod \"e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:07:43.973783 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.973763 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-workload-certs\") pod \"e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:07:43.973830 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.973794 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istio-token\") pod \"e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:07:43.973884 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.973867 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4kch\" (UniqueName: \"kubernetes.io/projected/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-kube-api-access-f4kch\") pod \"e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:07:43.973920 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:43.973902 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-workload-socket\") pod \"e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:07:44.073941 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.073903 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr"] Oct 30 07:07:44.074492 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.074471 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istio-data\") pod \"e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:07:44.074606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.074514 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istio-podinfo\") pod \"e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:07:44.074606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.074542 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istiod-ca-cert\") pod \"e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:07:44.074720 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.074610 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-workload-certs\") pod \"e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:07:44.074720 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.074642 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istio-token\") pod \"e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:07:44.074720 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.074704 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4kch\" (UniqueName: \"kubernetes.io/projected/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-kube-api-access-f4kch\") pod \"e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:07:44.074858 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.074744 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-workload-socket\") pod \"e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:07:44.074858 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.074782 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istio-envoy\") pod \"e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:07:44.074858 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.074840 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-credential-socket\") pod \"e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:07:44.075011 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.074936 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istio-data\") pod \"e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:07:44.075256 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.075227 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istiod-ca-cert\") pod \"e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:07:44.075549 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.075522 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-credential-socket\") pod \"e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:07:44.075675 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.075547 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-workload-certs\") pod \"e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:07:44.075764 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.075673 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-workload-socket\") pod \"e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:07:44.077322 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:07:44.077294 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2895d955_4a1f_45f1_94cb_251e4996c772.slice/crio-7b9eeb9a2fe0b25811586d9a5dd7b63d40f2a39933010af0bc6c6455f7bd7eeb WatchSource:0}: Error finding container 7b9eeb9a2fe0b25811586d9a5dd7b63d40f2a39933010af0bc6c6455f7bd7eeb: Status 404 returned error can't find the container with id 7b9eeb9a2fe0b25811586d9a5dd7b63d40f2a39933010af0bc6c6455f7bd7eeb Oct 30 07:07:44.077940 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.077912 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istio-envoy\") pod \"e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:07:44.078355 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.078259 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istio-podinfo\") pod \"e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:07:44.080235 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.080201 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:07:44.080314 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.080298 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:07:44.080351 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.080341 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:07:44.098194 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.098149 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istio-token\") pod \"e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:07:44.098419 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.098393 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4kch\" (UniqueName: \"kubernetes.io/projected/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-kube-api-access-f4kch\") pod \"e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:07:44.215454 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.215417 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:07:44.531932 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.531903 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx"] Oct 30 07:07:44.535137 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:07:44.535101 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c2b8f1_1449_4a94_a13e_cee4479e4d7c.slice/crio-68448250867e3afb459ef727da6b0fca3990cb9174df6d435b2e4d4622f617df WatchSource:0}: Error finding container 68448250867e3afb459ef727da6b0fca3990cb9174df6d435b2e4d4622f617df: Status 404 returned error can't find the container with id 68448250867e3afb459ef727da6b0fca3990cb9174df6d435b2e4d4622f617df Oct 30 07:07:44.537504 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.537473 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:07:44.537611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.537555 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:07:44.537660 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.537617 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:07:44.635366 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.635320 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" event={"ID":"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c","Type":"ContainerStarted","Data":"35010d24684ad0915687dc73daa10898db59dc7ae8b822a437a254c0f83783a2"} Oct 30 07:07:44.635524 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.635378 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" event={"ID":"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c","Type":"ContainerStarted","Data":"68448250867e3afb459ef727da6b0fca3990cb9174df6d435b2e4d4622f617df"} Oct 30 07:07:44.637381 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.637350 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" event={"ID":"2895d955-4a1f-45f1-94cb-251e4996c772","Type":"ContainerStarted","Data":"1556c961a76426de9668eef058d93a69e4dc81cf00f9dfb6203cd2334dd6dcb8"} Oct 30 07:07:44.637502 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.637406 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" event={"ID":"2895d955-4a1f-45f1-94cb-251e4996c772","Type":"ContainerStarted","Data":"7b9eeb9a2fe0b25811586d9a5dd7b63d40f2a39933010af0bc6c6455f7bd7eeb"} Oct 30 07:07:44.639458 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.639385 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" event={"ID":"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323","Type":"ContainerStarted","Data":"b6c985b661a41ddfcfffdb4522a7817ef37efc8f762b02ca841189f23d6108cf"} Oct 30 07:07:44.639458 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.639434 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" event={"ID":"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323","Type":"ContainerStarted","Data":"b5cc6811ef408ff64fd9f6a630c2af2f0e40b2a2269e6c58178a86f48e90e724"} Oct 30 07:07:44.680975 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.680917 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" podStartSLOduration=1.6808978620000001 podStartE2EDuration="1.680897862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:07:44.678719233 +0000 UTC m=+671.021767199" watchObservedRunningTime="2026-10-30 07:07:44.680897862 +0000 UTC m=+671.023945827" Oct 30 07:07:44.723891 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.723847 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" podStartSLOduration=1.723832293 podStartE2EDuration="1.723832293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:07:44.721305909 +0000 UTC m=+671.064353899" watchObservedRunningTime="2026-10-30 07:07:44.723832293 +0000 UTC m=+671.066880257" Oct 30 07:07:44.761077 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.761042 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:07:44.764541 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:44.764492 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" podStartSLOduration=1.764478607 podStartE2EDuration="1.764478607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:07:44.763645039 +0000 UTC m=+671.106693016" watchObservedRunningTime="2026-10-30 07:07:44.764478607 +0000 UTC m=+671.107526571" Oct 30 07:07:45.216490 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.216451 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:07:45.222473 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.222441 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:07:45.227235 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.227191 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq"] Oct 30 07:07:45.231828 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.231798 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:07:45.246946 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.246473 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-37f3d312-openshift-default-dockercfg-k74c5\"" Oct 30 07:07:45.255606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.255536 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq"] Oct 30 07:07:45.295381 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.295344 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-workload-socket\") pod \"e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:07:45.295381 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.295384 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-istio-envoy\") pod \"e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:07:45.295682 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.295479 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-credential-socket\") pod \"e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:07:45.295682 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.295520 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-workload-certs\") pod \"e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:07:45.295682 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.295612 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342054a0-c94d-48c6-8041-edccd3763e1b-istiod-ca-cert\") pod \"e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:07:45.295682 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.295640 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-istio-data\") pod \"e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:07:45.295897 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.295738 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342054a0-c94d-48c6-8041-edccd3763e1b-istio-podinfo\") pod \"e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:07:45.295897 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.295758 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/342054a0-c94d-48c6-8041-edccd3763e1b-istio-token\") pod \"e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:07:45.295897 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.295787 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dpcx\" (UniqueName: \"kubernetes.io/projected/342054a0-c94d-48c6-8041-edccd3763e1b-kube-api-access-9dpcx\") pod \"e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:07:45.402155 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.397024 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-workload-socket\") pod \"e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:07:45.402155 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.397520 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-workload-socket\") pod \"e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:07:45.402155 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.397628 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-istio-envoy\") pod \"e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:07:45.402155 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.397975 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-credential-socket\") pod \"e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:07:45.402155 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.397999 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-workload-certs\") pod \"e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:07:45.402155 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.398024 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342054a0-c94d-48c6-8041-edccd3763e1b-istiod-ca-cert\") pod \"e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:07:45.402155 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.398226 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-istio-data\") pod \"e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:07:45.402155 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.398555 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342054a0-c94d-48c6-8041-edccd3763e1b-istio-podinfo\") pod \"e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:07:45.402155 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.398715 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/342054a0-c94d-48c6-8041-edccd3763e1b-istio-token\") pod \"e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:07:45.402155 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.398800 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342054a0-c94d-48c6-8041-edccd3763e1b-istiod-ca-cert\") pod \"e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:07:45.402155 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.398501 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-credential-socket\") pod \"e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:07:45.402155 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.398865 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dpcx\" (UniqueName: \"kubernetes.io/projected/342054a0-c94d-48c6-8041-edccd3763e1b-kube-api-access-9dpcx\") pod \"e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:07:45.402155 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.399589 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-workload-certs\") pod \"e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:07:45.402155 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.399811 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-istio-data\") pod \"e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:07:45.402155 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.400337 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-istio-envoy\") pod \"e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:07:45.405122 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.405075 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342054a0-c94d-48c6-8041-edccd3763e1b-istio-podinfo\") pod \"e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:07:45.421652 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.421566 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/342054a0-c94d-48c6-8041-edccd3763e1b-istio-token\") pod \"e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:07:45.421828 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.421804 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dpcx\" (UniqueName: \"kubernetes.io/projected/342054a0-c94d-48c6-8041-edccd3763e1b-kube-api-access-9dpcx\") pod \"e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:07:45.549842 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.549762 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:07:45.618683 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.616770 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:07:45.622777 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.622749 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:07:45.644999 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.644917 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:07:45.644999 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.644965 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:07:45.645670 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.645647 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:07:45.645750 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.645719 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:07:45.771491 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.771466 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:07:45.871035 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.868829 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq"] Oct 30 07:07:45.876104 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:07:45.876072 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod342054a0_c94d_48c6_8041_edccd3763e1b.slice/crio-1cec421bd4458f16673f99570ed70bb796f397ca7267c0b7851bc42e746a921b WatchSource:0}: Error finding container 1cec421bd4458f16673f99570ed70bb796f397ca7267c0b7851bc42e746a921b: Status 404 returned error can't find the container with id 1cec421bd4458f16673f99570ed70bb796f397ca7267c0b7851bc42e746a921b Oct 30 07:07:45.878238 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.878205 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:07:45.878347 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.878282 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:07:45.878347 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:45.878324 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:07:46.650849 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:46.650808 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" event={"ID":"342054a0-c94d-48c6-8041-edccd3763e1b","Type":"ContainerStarted","Data":"f748aee0e36fbb055f22676f040610fa3a85d471a8eacb1e2b20ae379b8d5f59"} Oct 30 07:07:46.651274 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:46.650877 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" event={"ID":"342054a0-c94d-48c6-8041-edccd3763e1b","Type":"ContainerStarted","Data":"1cec421bd4458f16673f99570ed70bb796f397ca7267c0b7851bc42e746a921b"} Oct 30 07:07:46.651545 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:46.651524 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:07:46.652475 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:46.652455 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:07:46.700194 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:46.700121 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" podStartSLOduration=1.700103455 podStartE2EDuration="1.700103455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:07:46.699404287 +0000 UTC m=+673.042452285" watchObservedRunningTime="2026-10-30 07:07:46.700103455 +0000 UTC m=+673.043151421" Oct 30 07:07:47.550743 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:47.550703 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:07:47.556698 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:47.556670 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:07:47.657243 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:47.657209 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:07:47.658134 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:47.658113 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:07:51.192334 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:51.192294 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx"] Oct 30 07:07:51.196159 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:51.196132 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx" Oct 30 07:07:51.212214 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:51.212180 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-a5d88019-serving-cert\"" Oct 30 07:07:51.225324 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:51.225290 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx"] Oct 30 07:07:51.294243 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:51.294204 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee7d77ec-eae3-4640-8688-511efda613e4-maas-api-tls\") pod \"maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx\" (UID: \"ee7d77ec-eae3-4640-8688-511efda613e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx" Oct 30 07:07:51.294423 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:51.294252 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee7d77ec-eae3-4640-8688-511efda613e4-metrics-certs\") pod \"maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx\" (UID: \"ee7d77ec-eae3-4640-8688-511efda613e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx" Oct 30 07:07:51.294423 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:51.294322 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee7d77ec-eae3-4640-8688-511efda613e4-openshift-service-ca\") pod \"maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx\" (UID: \"ee7d77ec-eae3-4640-8688-511efda613e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx" Oct 30 07:07:51.294512 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:51.294423 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czvzs\" (UniqueName: \"kubernetes.io/projected/ee7d77ec-eae3-4640-8688-511efda613e4-kube-api-access-czvzs\") pod \"maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx\" (UID: \"ee7d77ec-eae3-4640-8688-511efda613e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx" Oct 30 07:07:51.395969 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:51.395926 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee7d77ec-eae3-4640-8688-511efda613e4-maas-api-tls\") pod \"maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx\" (UID: \"ee7d77ec-eae3-4640-8688-511efda613e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx" Oct 30 07:07:51.396172 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:51.395975 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee7d77ec-eae3-4640-8688-511efda613e4-metrics-certs\") pod \"maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx\" (UID: \"ee7d77ec-eae3-4640-8688-511efda613e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx" Oct 30 07:07:51.396172 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:51.396008 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee7d77ec-eae3-4640-8688-511efda613e4-openshift-service-ca\") pod \"maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx\" (UID: \"ee7d77ec-eae3-4640-8688-511efda613e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx" Oct 30 07:07:51.396172 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:51.396052 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czvzs\" (UniqueName: \"kubernetes.io/projected/ee7d77ec-eae3-4640-8688-511efda613e4-kube-api-access-czvzs\") pod \"maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx\" (UID: \"ee7d77ec-eae3-4640-8688-511efda613e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx" Oct 30 07:07:51.396172 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:07:51.396076 4951 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-a5d88019-serving-cert: secret "maas-api-e2e-migrate-a5d88019-serving-cert" not found Oct 30 07:07:51.396172 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:07:51.396152 4951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee7d77ec-eae3-4640-8688-511efda613e4-maas-api-tls podName:ee7d77ec-eae3-4640-8688-511efda613e4 nodeName:}" failed. No retries permitted until 2026-10-30 07:07:51.896135907 +0000 UTC m=+678.239183863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ee7d77ec-eae3-4640-8688-511efda613e4-maas-api-tls") pod "maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx" (UID: "ee7d77ec-eae3-4640-8688-511efda613e4") : secret "maas-api-e2e-migrate-a5d88019-serving-cert" not found Oct 30 07:07:51.396895 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:51.396870 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee7d77ec-eae3-4640-8688-511efda613e4-openshift-service-ca\") pod \"maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx\" (UID: \"ee7d77ec-eae3-4640-8688-511efda613e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx" Oct 30 07:07:51.398521 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:51.398504 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee7d77ec-eae3-4640-8688-511efda613e4-metrics-certs\") pod \"maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx\" (UID: \"ee7d77ec-eae3-4640-8688-511efda613e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx" Oct 30 07:07:51.421171 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:51.421122 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czvzs\" (UniqueName: \"kubernetes.io/projected/ee7d77ec-eae3-4640-8688-511efda613e4-kube-api-access-czvzs\") pod \"maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx\" (UID: \"ee7d77ec-eae3-4640-8688-511efda613e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx" Oct 30 07:07:51.900737 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:51.900696 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee7d77ec-eae3-4640-8688-511efda613e4-maas-api-tls\") pod \"maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx\" (UID: \"ee7d77ec-eae3-4640-8688-511efda613e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx" Oct 30 07:07:51.903158 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:51.903130 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee7d77ec-eae3-4640-8688-511efda613e4-maas-api-tls\") pod \"maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx\" (UID: \"ee7d77ec-eae3-4640-8688-511efda613e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx" Oct 30 07:07:52.110631 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:52.110590 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx" Oct 30 07:07:52.416468 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:52.416438 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx"] Oct 30 07:07:52.418495 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:07:52.418460 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee7d77ec_eae3_4640_8688_511efda613e4.slice/crio-ca316d3bffda028c975496f08814f1ce446da7805523b82ce49df6de7db444d2 WatchSource:0}: Error finding container ca316d3bffda028c975496f08814f1ce446da7805523b82ce49df6de7db444d2: Status 404 returned error can't find the container with id ca316d3bffda028c975496f08814f1ce446da7805523b82ce49df6de7db444d2 Oct 30 07:07:52.420071 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:52.420044 4951 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 07:07:52.699355 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:52.699272 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx" event={"ID":"ee7d77ec-eae3-4640-8688-511efda613e4","Type":"ContainerStarted","Data":"ca316d3bffda028c975496f08814f1ce446da7805523b82ce49df6de7db444d2"} Oct 30 07:07:52.894963 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:52.894929 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm"] Oct 30 07:07:52.898499 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:52.898475 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm" Oct 30 07:07:52.920728 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:52.918936 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-37f3d312-serving-cert\"" Oct 30 07:07:52.935618 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:52.932726 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm"] Oct 30 07:07:53.015760 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:53.015729 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8gdq\" (UniqueName: \"kubernetes.io/projected/1e44a6d7-6cfd-4fa1-ab31-caef4639842c-kube-api-access-d8gdq\") pod \"maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm\" (UID: \"1e44a6d7-6cfd-4fa1-ab31-caef4639842c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm" Oct 30 07:07:53.015925 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:53.015800 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e44a6d7-6cfd-4fa1-ab31-caef4639842c-metrics-certs\") pod \"maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm\" (UID: \"1e44a6d7-6cfd-4fa1-ab31-caef4639842c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm" Oct 30 07:07:53.015925 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:53.015917 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e44a6d7-6cfd-4fa1-ab31-caef4639842c-maas-api-tls\") pod \"maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm\" (UID: \"1e44a6d7-6cfd-4fa1-ab31-caef4639842c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm" Oct 30 07:07:53.016035 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:53.015951 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e44a6d7-6cfd-4fa1-ab31-caef4639842c-openshift-service-ca\") pod \"maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm\" (UID: \"1e44a6d7-6cfd-4fa1-ab31-caef4639842c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm" Oct 30 07:07:53.022785 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:07:53.022754 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:07:53.128594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:53.120178 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e44a6d7-6cfd-4fa1-ab31-caef4639842c-metrics-certs\") pod \"maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm\" (UID: \"1e44a6d7-6cfd-4fa1-ab31-caef4639842c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm" Oct 30 07:07:53.128594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:53.120245 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e44a6d7-6cfd-4fa1-ab31-caef4639842c-maas-api-tls\") pod \"maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm\" (UID: \"1e44a6d7-6cfd-4fa1-ab31-caef4639842c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm" Oct 30 07:07:53.128594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:53.120298 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e44a6d7-6cfd-4fa1-ab31-caef4639842c-openshift-service-ca\") pod \"maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm\" (UID: \"1e44a6d7-6cfd-4fa1-ab31-caef4639842c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm" Oct 30 07:07:53.128594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:53.120373 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8gdq\" (UniqueName: \"kubernetes.io/projected/1e44a6d7-6cfd-4fa1-ab31-caef4639842c-kube-api-access-d8gdq\") pod \"maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm\" (UID: \"1e44a6d7-6cfd-4fa1-ab31-caef4639842c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm" Oct 30 07:07:53.143595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:53.133038 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e44a6d7-6cfd-4fa1-ab31-caef4639842c-openshift-service-ca\") pod \"maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm\" (UID: \"1e44a6d7-6cfd-4fa1-ab31-caef4639842c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm" Oct 30 07:07:53.170591 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:53.153143 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e44a6d7-6cfd-4fa1-ab31-caef4639842c-maas-api-tls\") pod \"maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm\" (UID: \"1e44a6d7-6cfd-4fa1-ab31-caef4639842c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm" Oct 30 07:07:53.170591 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:53.161045 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e44a6d7-6cfd-4fa1-ab31-caef4639842c-metrics-certs\") pod \"maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm\" (UID: \"1e44a6d7-6cfd-4fa1-ab31-caef4639842c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm" Oct 30 07:07:53.183629 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:53.177938 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8gdq\" (UniqueName: \"kubernetes.io/projected/1e44a6d7-6cfd-4fa1-ab31-caef4639842c-kube-api-access-d8gdq\") pod \"maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm\" (UID: \"1e44a6d7-6cfd-4fa1-ab31-caef4639842c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm" Oct 30 07:07:53.214657 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:53.214627 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm" Oct 30 07:07:53.527219 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:53.527193 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm"] Oct 30 07:07:53.530162 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:07:53.530099 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e44a6d7_6cfd_4fa1_ab31_caef4639842c.slice/crio-452452d7f3cb7a6c612cbd83164dda25a07404e3e9fe407013391061715c7c36 WatchSource:0}: Error finding container 452452d7f3cb7a6c612cbd83164dda25a07404e3e9fe407013391061715c7c36: Status 404 returned error can't find the container with id 452452d7f3cb7a6c612cbd83164dda25a07404e3e9fe407013391061715c7c36 Oct 30 07:07:53.706622 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:53.706562 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm" event={"ID":"1e44a6d7-6cfd-4fa1-ab31-caef4639842c","Type":"ContainerStarted","Data":"452452d7f3cb7a6c612cbd83164dda25a07404e3e9fe407013391061715c7c36"} Oct 30 07:07:53.708016 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:53.707992 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx" event={"ID":"ee7d77ec-eae3-4640-8688-511efda613e4","Type":"ContainerStarted","Data":"ed23f7f821f9e018175c071242d61ab443d08384757e84f4a3e986128b9d771a"} Oct 30 07:07:53.708128 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:53.708121 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx" Oct 30 07:07:53.749547 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:53.749500 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx" podStartSLOduration=2.3567929 podStartE2EDuration="2.749486674s" totalImagesPullingTime="392.693774ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:07:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:07:53.749051936 +0000 UTC m=+680.092099894" watchObservedRunningTime="2026-10-30 07:07:53.749486674 +0000 UTC m=+680.092534638" Oct 30 07:07:54.715129 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:54.715091 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm" event={"ID":"1e44a6d7-6cfd-4fa1-ab31-caef4639842c","Type":"ContainerStarted","Data":"561e579cf71b5a17431fe29a5e60d7840ff2983f3000e5053bf4b02b754e4f4d"} Oct 30 07:07:54.715541 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:54.715177 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm" Oct 30 07:07:54.756552 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:54.756484 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm" podStartSLOduration=2.360475632 podStartE2EDuration="2.756466864s" totalImagesPullingTime="395.991232ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:07:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:07:54.754994037 +0000 UTC m=+681.098042004" watchObservedRunningTime="2026-10-30 07:07:54.756466864 +0000 UTC m=+681.099514844" Oct 30 07:07:59.721101 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:07:59.721062 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx" Oct 30 07:08:00.752118 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:00.752083 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm" Oct 30 07:08:01.691022 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:01.690802 4951 patch_prober.go:29] interesting pod/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:08:01.691022 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:01.690827 4951 patch_prober.go:29] interesting pod/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:08:01.691022 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:01.690870 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" podUID="d3c2b8f1-1449-4a94-a13e-cee4479e4d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:08:01.691022 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:01.690913 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" podUID="8e2d9f9d-3fe2-4abe-8136-f34cdf8de323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:08:02.874859 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:02.872050 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc"] Oct 30 07:08:02.893809 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:02.883904 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc" Oct 30 07:08:02.899766 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:02.899645 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc"] Oct 30 07:08:02.916855 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:02.916803 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-77bbf25a-serving-cert\"" Oct 30 07:08:02.997642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:02.994305 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4kfx\" (UniqueName: \"kubernetes.io/projected/6a3b7d91-481b-453f-895e-0bd0a1799035-kube-api-access-c4kfx\") pod \"maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc\" (UID: \"6a3b7d91-481b-453f-895e-0bd0a1799035\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc" Oct 30 07:08:02.997642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:02.994376 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a3b7d91-481b-453f-895e-0bd0a1799035-metrics-certs\") pod \"maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc\" (UID: \"6a3b7d91-481b-453f-895e-0bd0a1799035\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc" Oct 30 07:08:02.997642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:02.994464 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a3b7d91-481b-453f-895e-0bd0a1799035-openshift-service-ca\") pod \"maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc\" (UID: \"6a3b7d91-481b-453f-895e-0bd0a1799035\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc" Oct 30 07:08:02.997642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:02.994563 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a3b7d91-481b-453f-895e-0bd0a1799035-maas-api-tls\") pod \"maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc\" (UID: \"6a3b7d91-481b-453f-895e-0bd0a1799035\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc" Oct 30 07:08:03.100595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:03.097390 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a3b7d91-481b-453f-895e-0bd0a1799035-openshift-service-ca\") pod \"maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc\" (UID: \"6a3b7d91-481b-453f-895e-0bd0a1799035\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc" Oct 30 07:08:03.100595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:03.097495 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a3b7d91-481b-453f-895e-0bd0a1799035-maas-api-tls\") pod \"maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc\" (UID: \"6a3b7d91-481b-453f-895e-0bd0a1799035\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc" Oct 30 07:08:03.100595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:03.097556 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4kfx\" (UniqueName: \"kubernetes.io/projected/6a3b7d91-481b-453f-895e-0bd0a1799035-kube-api-access-c4kfx\") pod \"maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc\" (UID: \"6a3b7d91-481b-453f-895e-0bd0a1799035\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc" Oct 30 07:08:03.100595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:03.098356 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a3b7d91-481b-453f-895e-0bd0a1799035-openshift-service-ca\") pod \"maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc\" (UID: \"6a3b7d91-481b-453f-895e-0bd0a1799035\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc" Oct 30 07:08:03.100595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:03.098821 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a3b7d91-481b-453f-895e-0bd0a1799035-metrics-certs\") pod \"maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc\" (UID: \"6a3b7d91-481b-453f-895e-0bd0a1799035\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc" Oct 30 07:08:03.119937 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:03.119359 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a3b7d91-481b-453f-895e-0bd0a1799035-metrics-certs\") pod \"maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc\" (UID: \"6a3b7d91-481b-453f-895e-0bd0a1799035\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc" Oct 30 07:08:03.137429 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:03.137322 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a3b7d91-481b-453f-895e-0bd0a1799035-maas-api-tls\") pod \"maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc\" (UID: \"6a3b7d91-481b-453f-895e-0bd0a1799035\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc" Oct 30 07:08:03.142366 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:03.142297 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4kfx\" (UniqueName: \"kubernetes.io/projected/6a3b7d91-481b-453f-895e-0bd0a1799035-kube-api-access-c4kfx\") pod \"maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc\" (UID: \"6a3b7d91-481b-453f-895e-0bd0a1799035\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc" Oct 30 07:08:03.207400 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:03.207097 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc" Oct 30 07:08:03.598332 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:03.598201 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc"] Oct 30 07:08:03.601687 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:08:03.601655 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a3b7d91_481b_453f_895e_0bd0a1799035.slice/crio-074273167b4183d2ee7c1813de90f1d238892ee5396add37454c2ed4abec16c3 WatchSource:0}: Error finding container 074273167b4183d2ee7c1813de90f1d238892ee5396add37454c2ed4abec16c3: Status 404 returned error can't find the container with id 074273167b4183d2ee7c1813de90f1d238892ee5396add37454c2ed4abec16c3 Oct 30 07:08:03.884063 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:03.883972 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc" event={"ID":"6a3b7d91-481b-453f-895e-0bd0a1799035","Type":"ContainerStarted","Data":"074273167b4183d2ee7c1813de90f1d238892ee5396add37454c2ed4abec16c3"} Oct 30 07:08:04.891773 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:04.891734 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc" event={"ID":"6a3b7d91-481b-453f-895e-0bd0a1799035","Type":"ContainerStarted","Data":"680150a1aab59880841e9f9ecafaaca569e7707d4a1831e9e5c9038f75114a7b"} Oct 30 07:08:04.892304 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:04.891815 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc" Oct 30 07:08:07.910179 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:07.910141 4951 generic.go:356] "Generic (PLEG): container finished" podID="d3c2b8f1-1449-4a94-a13e-cee4479e4d7c" containerID="35010d24684ad0915687dc73daa10898db59dc7ae8b822a437a254c0f83783a2" exitCode=0 Oct 30 07:08:07.911447 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:07.910223 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" event={"ID":"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c","Type":"ContainerDied","Data":"35010d24684ad0915687dc73daa10898db59dc7ae8b822a437a254c0f83783a2"} Oct 30 07:08:07.911447 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:07.910745 4951 scope.go:122] "RemoveContainer" containerID="35010d24684ad0915687dc73daa10898db59dc7ae8b822a437a254c0f83783a2" Oct 30 07:08:07.912251 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:07.912223 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:08:07.912314 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:07.912292 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:08:07.912359 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:07.912324 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:08:07.994685 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:07.993729 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc" podStartSLOduration=5.428572103 podStartE2EDuration="5.993709641s" totalImagesPullingTime="565.137538ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:08:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:08:04.940408758 +0000 UTC m=+691.283456724" watchObservedRunningTime="2026-10-30 07:08:07.993709641 +0000 UTC m=+694.336757607" Oct 30 07:08:08.919329 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:08.919285 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" event={"ID":"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c","Type":"ContainerStarted","Data":"a7df57889e88bac8af552482152c08d050e9b4d77f1964a72b7a64f54cabd517"} Oct 30 07:08:08.921281 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:08.921240 4951 generic.go:356] "Generic (PLEG): container finished" podID="8e2d9f9d-3fe2-4abe-8136-f34cdf8de323" containerID="b6c985b661a41ddfcfffdb4522a7817ef37efc8f762b02ca841189f23d6108cf" exitCode=0 Oct 30 07:08:08.921435 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:08.921298 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" event={"ID":"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323","Type":"ContainerDied","Data":"b6c985b661a41ddfcfffdb4522a7817ef37efc8f762b02ca841189f23d6108cf"} Oct 30 07:08:08.927595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:08.921723 4951 scope.go:122] "RemoveContainer" containerID="b6c985b661a41ddfcfffdb4522a7817ef37efc8f762b02ca841189f23d6108cf" Oct 30 07:08:08.927595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:08.927308 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:08:08.927595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:08.927408 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:08:08.927595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:08.927458 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:08:09.216534 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:09.216427 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:08:09.216911 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:09.216864 4951 patch_prober.go:29] interesting pod/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 30 07:08:09.217060 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:09.216923 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" podUID="d3c2b8f1-1449-4a94-a13e-cee4479e4d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 30 07:08:09.975594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:09.975169 4951 generic.go:356] "Generic (PLEG): container finished" podID="342054a0-c94d-48c6-8041-edccd3763e1b" containerID="f748aee0e36fbb055f22676f040610fa3a85d471a8eacb1e2b20ae379b8d5f59" exitCode=0 Oct 30 07:08:09.975594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:09.975320 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" event={"ID":"342054a0-c94d-48c6-8041-edccd3763e1b","Type":"ContainerDied","Data":"f748aee0e36fbb055f22676f040610fa3a85d471a8eacb1e2b20ae379b8d5f59"} Oct 30 07:08:09.976448 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:09.976423 4951 scope.go:122] "RemoveContainer" containerID="f748aee0e36fbb055f22676f040610fa3a85d471a8eacb1e2b20ae379b8d5f59" Oct 30 07:08:09.979598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:09.979442 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:08:09.979598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:09.979541 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:08:09.979788 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:09.979613 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:08:09.984947 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:09.984857 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" event={"ID":"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323","Type":"ContainerStarted","Data":"b63861a388d8570d3f07e13ffb9b28d092d4a148c81e4cd1038c48b8b3373d1c"} Oct 30 07:08:10.216289 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:10.216262 4951 patch_prober.go:29] interesting pod/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 30 07:08:10.216400 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:10.216308 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" podUID="d3c2b8f1-1449-4a94-a13e-cee4479e4d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 30 07:08:10.620455 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:10.620419 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:08:10.620672 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:10.620537 4951 patch_prober.go:29] interesting pod/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 30 07:08:10.620672 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:10.620609 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" podUID="8e2d9f9d-3fe2-4abe-8136-f34cdf8de323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 30 07:08:10.908680 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:10.906789 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc" Oct 30 07:08:11.011607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:11.010258 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" event={"ID":"342054a0-c94d-48c6-8041-edccd3763e1b","Type":"ContainerStarted","Data":"8790e1ff280218789a67887928d6071aeaccbc0d89c9f88ca7b1c4480aa3f434"} Oct 30 07:08:11.216135 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:11.216058 4951 patch_prober.go:29] interesting pod/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 30 07:08:11.216999 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:11.216965 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" podUID="d3c2b8f1-1449-4a94-a13e-cee4479e4d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 30 07:08:11.551066 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:11.551027 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:08:11.551478 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:11.551385 4951 patch_prober.go:29] interesting pod/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 30 07:08:11.551673 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:11.551648 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" podUID="342054a0-c94d-48c6-8041-edccd3763e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 30 07:08:11.616364 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:11.616323 4951 patch_prober.go:29] interesting pod/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 30 07:08:11.616649 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:11.616618 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" podUID="8e2d9f9d-3fe2-4abe-8136-f34cdf8de323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 30 07:08:12.216787 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:12.216341 4951 patch_prober.go:29] interesting pod/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 30 07:08:12.216787 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:12.216393 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" podUID="d3c2b8f1-1449-4a94-a13e-cee4479e4d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 30 07:08:12.554593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:12.553975 4951 patch_prober.go:29] interesting pod/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 30 07:08:12.554593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:12.554039 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" podUID="342054a0-c94d-48c6-8041-edccd3763e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 30 07:08:12.616404 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:12.616369 4951 patch_prober.go:29] interesting pod/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 30 07:08:12.616638 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:12.616613 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" podUID="8e2d9f9d-3fe2-4abe-8136-f34cdf8de323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 30 07:08:13.229600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:13.222695 4951 patch_prober.go:29] interesting pod/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 30 07:08:13.229600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:13.222759 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" podUID="d3c2b8f1-1449-4a94-a13e-cee4479e4d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 30 07:08:13.561510 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:13.561006 4951 patch_prober.go:29] interesting pod/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 30 07:08:13.561510 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:13.561074 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" podUID="342054a0-c94d-48c6-8041-edccd3763e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 30 07:08:13.616517 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:13.616479 4951 patch_prober.go:29] interesting pod/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 30 07:08:13.616748 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:13.616547 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" podUID="8e2d9f9d-3fe2-4abe-8136-f34cdf8de323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 30 07:08:14.216127 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:14.216093 4951 patch_prober.go:29] interesting pod/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 30 07:08:14.216359 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:14.216336 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" podUID="d3c2b8f1-1449-4a94-a13e-cee4479e4d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 30 07:08:14.565375 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:14.564908 4951 patch_prober.go:29] interesting pod/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 30 07:08:14.565375 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:14.564970 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" podUID="342054a0-c94d-48c6-8041-edccd3763e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 30 07:08:14.616435 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:14.615984 4951 patch_prober.go:29] interesting pod/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 30 07:08:14.616435 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:14.616051 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" podUID="8e2d9f9d-3fe2-4abe-8136-f34cdf8de323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 30 07:08:15.221755 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:15.221721 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:08:15.240738 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:15.240633 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:08:15.257417 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:15.256534 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:08:15.551406 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:15.550971 4951 patch_prober.go:29] interesting pod/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 30 07:08:15.551406 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:15.551033 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" podUID="342054a0-c94d-48c6-8041-edccd3763e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 30 07:08:15.621523 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:15.621496 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:08:15.629689 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:15.629618 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:08:15.648341 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:15.648308 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:08:16.551002 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:16.550969 4951 patch_prober.go:29] interesting pod/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 30 07:08:16.551216 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:16.551022 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" podUID="342054a0-c94d-48c6-8041-edccd3763e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 30 07:08:16.653595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:16.653475 4951 patch_prober.go:29] interesting pod/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 30 07:08:16.653595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:16.653535 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" podUID="2895d955-4a1f-45f1-94cb-251e4996c772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 30 07:08:17.121239 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:17.121203 4951 generic.go:356] "Generic (PLEG): container finished" podID="2895d955-4a1f-45f1-94cb-251e4996c772" containerID="1556c961a76426de9668eef058d93a69e4dc81cf00f9dfb6203cd2334dd6dcb8" exitCode=0 Oct 30 07:08:17.121416 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:17.121273 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" event={"ID":"2895d955-4a1f-45f1-94cb-251e4996c772","Type":"ContainerDied","Data":"1556c961a76426de9668eef058d93a69e4dc81cf00f9dfb6203cd2334dd6dcb8"} Oct 30 07:08:17.121914 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:17.121898 4951 scope.go:122] "RemoveContainer" containerID="1556c961a76426de9668eef058d93a69e4dc81cf00f9dfb6203cd2334dd6dcb8" Oct 30 07:08:17.123143 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:17.123109 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:08:17.123241 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:17.123199 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:08:17.123283 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:17.123248 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:08:17.557491 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:17.557456 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:08:17.558280 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:17.558247 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:08:17.559271 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:17.559252 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:08:18.149628 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:18.149587 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" event={"ID":"2895d955-4a1f-45f1-94cb-251e4996c772","Type":"ContainerStarted","Data":"30cd4c23e06b227f010dabcf29989b5ccc2450b3c721286c664db7f16953e307"} Oct 30 07:08:18.761076 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:18.761029 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:08:18.761347 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:18.761318 4951 patch_prober.go:29] interesting pod/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 30 07:08:18.761522 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:18.761497 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" podUID="2895d955-4a1f-45f1-94cb-251e4996c772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 30 07:08:19.348985 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:19.348949 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx"] Oct 30 07:08:19.349455 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:19.349204 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx" podUID="ee7d77ec-eae3-4640-8688-511efda613e4" containerName="maas-api" containerID="cri-o://ed23f7f821f9e018175c071242d61ab443d08384757e84f4a3e986128b9d771a" gracePeriod=30 Oct 30 07:08:19.684256 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:19.684021 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx" Oct 30 07:08:19.764541 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:19.762966 4951 patch_prober.go:29] interesting pod/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 30 07:08:19.764541 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:19.763022 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" podUID="2895d955-4a1f-45f1-94cb-251e4996c772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 30 07:08:19.765319 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:19.765282 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee7d77ec-eae3-4640-8688-511efda613e4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee7d77ec-eae3-4640-8688-511efda613e4-maas-api-tls\") pod \"ee7d77ec-eae3-4640-8688-511efda613e4\" (UID: \"ee7d77ec-eae3-4640-8688-511efda613e4\") " Oct 30 07:08:19.765566 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:19.765534 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee7d77ec-eae3-4640-8688-511efda613e4-kube-api-access-czvzs\" (UniqueName: \"kubernetes.io/projected/ee7d77ec-eae3-4640-8688-511efda613e4-kube-api-access-czvzs\") pod \"ee7d77ec-eae3-4640-8688-511efda613e4\" (UID: \"ee7d77ec-eae3-4640-8688-511efda613e4\") " Oct 30 07:08:19.765750 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:19.765731 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee7d77ec-eae3-4640-8688-511efda613e4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee7d77ec-eae3-4640-8688-511efda613e4-openshift-service-ca\") pod \"ee7d77ec-eae3-4640-8688-511efda613e4\" (UID: \"ee7d77ec-eae3-4640-8688-511efda613e4\") " Oct 30 07:08:19.765913 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:19.765896 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee7d77ec-eae3-4640-8688-511efda613e4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee7d77ec-eae3-4640-8688-511efda613e4-metrics-certs\") pod \"ee7d77ec-eae3-4640-8688-511efda613e4\" (UID: \"ee7d77ec-eae3-4640-8688-511efda613e4\") " Oct 30 07:08:19.766476 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:19.766158 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee7d77ec-eae3-4640-8688-511efda613e4-openshift-service-ca" pod "ee7d77ec-eae3-4640-8688-511efda613e4" (UID: "ee7d77ec-eae3-4640-8688-511efda613e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:08:19.769928 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:19.769901 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee7d77ec-eae3-4640-8688-511efda613e4-maas-api-tls" pod "ee7d77ec-eae3-4640-8688-511efda613e4" (UID: "ee7d77ec-eae3-4640-8688-511efda613e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:08:19.770221 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:19.770143 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee7d77ec-eae3-4640-8688-511efda613e4-metrics-certs" pod "ee7d77ec-eae3-4640-8688-511efda613e4" (UID: "ee7d77ec-eae3-4640-8688-511efda613e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:08:19.770221 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:19.770170 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee7d77ec-eae3-4640-8688-511efda613e4-kube-api-access-czvzs" pod "ee7d77ec-eae3-4640-8688-511efda613e4" (UID: "ee7d77ec-eae3-4640-8688-511efda613e4"). InnerVolumeSpecName "kube-api-access-czvzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:08:19.867126 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:19.867086 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee7d77ec-eae3-4640-8688-511efda613e4-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:19.867126 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:19.867115 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czvzs\" (UniqueName: \"kubernetes.io/projected/ee7d77ec-eae3-4640-8688-511efda613e4-kube-api-access-czvzs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:19.867126 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:19.867128 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee7d77ec-eae3-4640-8688-511efda613e4-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:19.867365 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:19.867141 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee7d77ec-eae3-4640-8688-511efda613e4-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:20.105881 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:20.105845 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx"] Oct 30 07:08:20.106334 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:20.106302 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" podUID="d3c2b8f1-1449-4a94-a13e-cee4479e4d7c" containerName="istio-proxy" containerID="cri-o://a7df57889e88bac8af552482152c08d050e9b4d77f1964a72b7a64f54cabd517" gracePeriod=30 Oct 30 07:08:20.174270 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:20.174233 4951 generic.go:356] "Generic (PLEG): container finished" podID="ee7d77ec-eae3-4640-8688-511efda613e4" containerID="ed23f7f821f9e018175c071242d61ab443d08384757e84f4a3e986128b9d771a" exitCode=0 Oct 30 07:08:20.174442 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:20.174325 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx" event={"ID":"ee7d77ec-eae3-4640-8688-511efda613e4","Type":"ContainerDied","Data":"ed23f7f821f9e018175c071242d61ab443d08384757e84f4a3e986128b9d771a"} Oct 30 07:08:20.174442 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:20.174385 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx" event={"ID":"ee7d77ec-eae3-4640-8688-511efda613e4","Type":"ContainerDied","Data":"ca316d3bffda028c975496f08814f1ce446da7805523b82ce49df6de7db444d2"} Oct 30 07:08:20.174442 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:20.174405 4951 scope.go:122] "RemoveContainer" containerID="ed23f7f821f9e018175c071242d61ab443d08384757e84f4a3e986128b9d771a" Oct 30 07:08:20.174537 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:20.174531 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx" Oct 30 07:08:20.184009 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:20.183975 4951 scope.go:122] "RemoveContainer" containerID="ed23f7f821f9e018175c071242d61ab443d08384757e84f4a3e986128b9d771a" Oct 30 07:08:20.184320 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:08:20.184297 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed23f7f821f9e018175c071242d61ab443d08384757e84f4a3e986128b9d771a\": container with ID starting with ed23f7f821f9e018175c071242d61ab443d08384757e84f4a3e986128b9d771a not found: ID does not exist" containerID="ed23f7f821f9e018175c071242d61ab443d08384757e84f4a3e986128b9d771a" Oct 30 07:08:20.184368 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:20.184331 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed23f7f821f9e018175c071242d61ab443d08384757e84f4a3e986128b9d771a"} err="failed to get container status \"ed23f7f821f9e018175c071242d61ab443d08384757e84f4a3e986128b9d771a\": rpc error: code = NotFound desc = could not find container \"ed23f7f821f9e018175c071242d61ab443d08384757e84f4a3e986128b9d771a\": container with ID starting with ed23f7f821f9e018175c071242d61ab443d08384757e84f4a3e986128b9d771a not found: ID does not exist" Oct 30 07:08:20.254600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:20.233997 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx"] Oct 30 07:08:20.258596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:20.257616 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a5d88019-666f584bf8-qxmmx"] Oct 30 07:08:20.769321 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:20.768401 4951 patch_prober.go:29] interesting pod/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 30 07:08:20.769321 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:20.768470 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" podUID="2895d955-4a1f-45f1-94cb-251e4996c772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 30 07:08:21.761294 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:21.761253 4951 patch_prober.go:29] interesting pod/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 30 07:08:21.761493 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:21.761312 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" podUID="2895d955-4a1f-45f1-94cb-251e4996c772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 30 07:08:21.769965 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:21.769935 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee7d77ec-eae3-4640-8688-511efda613e4" path="/var/lib/kubelet/pods/ee7d77ec-eae3-4640-8688-511efda613e4/volumes" Oct 30 07:08:22.768273 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:22.768238 4951 patch_prober.go:29] interesting pod/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 30 07:08:22.768669 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:22.768623 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" podUID="2895d955-4a1f-45f1-94cb-251e4996c772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 30 07:08:23.027771 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:08:23.027649 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:08:23.893375 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:23.893340 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:08:23.894089 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:23.894069 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:08:23.895004 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:23.894961 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:08:25.376787 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.376716 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:08:25.467969 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.467935 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-kube-api-access-f4kch\" (UniqueName: \"kubernetes.io/projected/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-kube-api-access-f4kch\") pod \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " Oct 30 07:08:25.468139 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.467998 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istiod-ca-cert\") pod \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " Oct 30 07:08:25.468139 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.468072 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-workload-certs\") pod \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " Oct 30 07:08:25.468139 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.468099 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istio-token\" (UniqueName: \"kubernetes.io/projected/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istio-token\") pod \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " Oct 30 07:08:25.468308 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.468173 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istio-data\") pod \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " Oct 30 07:08:25.468308 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.468212 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istio-podinfo\") pod \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " Oct 30 07:08:25.468308 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.468243 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-credential-socket\") pod \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " Oct 30 07:08:25.468308 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.468285 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-workload-socket\") pod \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " Oct 30 07:08:25.468549 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.468333 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istio-envoy\") pod \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\" (UID: \"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c\") " Oct 30 07:08:25.468549 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.468400 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-workload-certs" pod "d3c2b8f1-1449-4a94-a13e-cee4479e4d7c" (UID: "d3c2b8f1-1449-4a94-a13e-cee4479e4d7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:08:25.468549 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.468453 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istio-data" pod "d3c2b8f1-1449-4a94-a13e-cee4479e4d7c" (UID: "d3c2b8f1-1449-4a94-a13e-cee4479e4d7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:08:25.468549 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.468491 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istiod-ca-cert" pod "d3c2b8f1-1449-4a94-a13e-cee4479e4d7c" (UID: "d3c2b8f1-1449-4a94-a13e-cee4479e4d7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:08:25.468781 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.468650 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-credential-socket" pod "d3c2b8f1-1449-4a94-a13e-cee4479e4d7c" (UID: "d3c2b8f1-1449-4a94-a13e-cee4479e4d7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:08:25.468781 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.468692 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-workload-socket" pod "d3c2b8f1-1449-4a94-a13e-cee4479e4d7c" (UID: "d3c2b8f1-1449-4a94-a13e-cee4479e4d7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:08:25.468880 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.468862 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:25.468919 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.468887 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:25.468919 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.468901 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:25.468919 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.468915 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:25.469046 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.468928 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:25.470550 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.470523 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istio-token" pod "d3c2b8f1-1449-4a94-a13e-cee4479e4d7c" (UID: "d3c2b8f1-1449-4a94-a13e-cee4479e4d7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:08:25.470760 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.470741 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istio-envoy" pod "d3c2b8f1-1449-4a94-a13e-cee4479e4d7c" (UID: "d3c2b8f1-1449-4a94-a13e-cee4479e4d7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:08:25.470923 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.470814 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-kube-api-access-f4kch" pod "d3c2b8f1-1449-4a94-a13e-cee4479e4d7c" (UID: "d3c2b8f1-1449-4a94-a13e-cee4479e4d7c"). InnerVolumeSpecName "kube-api-access-f4kch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:08:25.470923 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.470821 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istio-podinfo" pod "d3c2b8f1-1449-4a94-a13e-cee4479e4d7c" (UID: "d3c2b8f1-1449-4a94-a13e-cee4479e4d7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:08:25.568597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.561778 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm"] Oct 30 07:08:25.568597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.562178 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm" podUID="1e44a6d7-6cfd-4fa1-ab31-caef4639842c" containerName="maas-api" containerID="cri-o://561e579cf71b5a17431fe29a5e60d7840ff2983f3000e5053bf4b02b754e4f4d" gracePeriod=30 Oct 30 07:08:25.571807 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.571708 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4kch\" (UniqueName: \"kubernetes.io/projected/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-kube-api-access-f4kch\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:25.571807 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.571746 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:25.571807 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.571762 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:25.571807 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.571776 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:25.721206 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.721168 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm" podUID="1e44a6d7-6cfd-4fa1-ab31-caef4639842c" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.48:8443/health\": dial tcp 10.132.0.48:8443: connect: connection refused" Oct 30 07:08:25.823386 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.823344 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm" Oct 30 07:08:25.877977 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.877937 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e44a6d7-6cfd-4fa1-ab31-caef4639842c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e44a6d7-6cfd-4fa1-ab31-caef4639842c-metrics-certs\") pod \"1e44a6d7-6cfd-4fa1-ab31-caef4639842c\" (UID: \"1e44a6d7-6cfd-4fa1-ab31-caef4639842c\") " Oct 30 07:08:25.878166 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.878147 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e44a6d7-6cfd-4fa1-ab31-caef4639842c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e44a6d7-6cfd-4fa1-ab31-caef4639842c-maas-api-tls\") pod \"1e44a6d7-6cfd-4fa1-ab31-caef4639842c\" (UID: \"1e44a6d7-6cfd-4fa1-ab31-caef4639842c\") " Oct 30 07:08:25.878242 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.878224 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e44a6d7-6cfd-4fa1-ab31-caef4639842c-kube-api-access-d8gdq\" (UniqueName: \"kubernetes.io/projected/1e44a6d7-6cfd-4fa1-ab31-caef4639842c-kube-api-access-d8gdq\") pod \"1e44a6d7-6cfd-4fa1-ab31-caef4639842c\" (UID: \"1e44a6d7-6cfd-4fa1-ab31-caef4639842c\") " Oct 30 07:08:25.878336 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.878317 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e44a6d7-6cfd-4fa1-ab31-caef4639842c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e44a6d7-6cfd-4fa1-ab31-caef4639842c-openshift-service-ca\") pod \"1e44a6d7-6cfd-4fa1-ab31-caef4639842c\" (UID: \"1e44a6d7-6cfd-4fa1-ab31-caef4639842c\") " Oct 30 07:08:25.878854 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.878828 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e44a6d7-6cfd-4fa1-ab31-caef4639842c-openshift-service-ca" pod "1e44a6d7-6cfd-4fa1-ab31-caef4639842c" (UID: "1e44a6d7-6cfd-4fa1-ab31-caef4639842c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:08:25.880670 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.880551 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e44a6d7-6cfd-4fa1-ab31-caef4639842c-maas-api-tls" pod "1e44a6d7-6cfd-4fa1-ab31-caef4639842c" (UID: "1e44a6d7-6cfd-4fa1-ab31-caef4639842c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:08:25.880775 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.880707 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e44a6d7-6cfd-4fa1-ab31-caef4639842c-metrics-certs" pod "1e44a6d7-6cfd-4fa1-ab31-caef4639842c" (UID: "1e44a6d7-6cfd-4fa1-ab31-caef4639842c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:08:25.880775 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.880760 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e44a6d7-6cfd-4fa1-ab31-caef4639842c-kube-api-access-d8gdq" pod "1e44a6d7-6cfd-4fa1-ab31-caef4639842c" (UID: "1e44a6d7-6cfd-4fa1-ab31-caef4639842c"). InnerVolumeSpecName "kube-api-access-d8gdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:08:25.979677 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.979566 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e44a6d7-6cfd-4fa1-ab31-caef4639842c-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:25.979677 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.979629 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e44a6d7-6cfd-4fa1-ab31-caef4639842c-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:25.979677 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.979642 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8gdq\" (UniqueName: \"kubernetes.io/projected/1e44a6d7-6cfd-4fa1-ab31-caef4639842c-kube-api-access-d8gdq\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:25.979677 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:25.979655 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e44a6d7-6cfd-4fa1-ab31-caef4639842c-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:26.000789 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:26.000759 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq"] Oct 30 07:08:26.000997 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:26.000974 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" podUID="8e2d9f9d-3fe2-4abe-8136-f34cdf8de323" containerName="istio-proxy" containerID="cri-o://b63861a388d8570d3f07e13ffb9b28d092d4a148c81e4cd1038c48b8b3373d1c" gracePeriod=30 Oct 30 07:08:26.219065 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:26.219029 4951 generic.go:356] "Generic (PLEG): container finished" podID="1e44a6d7-6cfd-4fa1-ab31-caef4639842c" containerID="561e579cf71b5a17431fe29a5e60d7840ff2983f3000e5053bf4b02b754e4f4d" exitCode=0 Oct 30 07:08:26.219223 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:26.219086 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm" Oct 30 07:08:26.219223 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:26.219121 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm" event={"ID":"1e44a6d7-6cfd-4fa1-ab31-caef4639842c","Type":"ContainerDied","Data":"561e579cf71b5a17431fe29a5e60d7840ff2983f3000e5053bf4b02b754e4f4d"} Oct 30 07:08:26.219223 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:26.219194 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm" event={"ID":"1e44a6d7-6cfd-4fa1-ab31-caef4639842c","Type":"ContainerDied","Data":"452452d7f3cb7a6c612cbd83164dda25a07404e3e9fe407013391061715c7c36"} Oct 30 07:08:26.219503 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:26.219227 4951 scope.go:122] "RemoveContainer" containerID="561e579cf71b5a17431fe29a5e60d7840ff2983f3000e5053bf4b02b754e4f4d" Oct 30 07:08:26.222932 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:26.222905 4951 generic.go:356] "Generic (PLEG): container finished" podID="d3c2b8f1-1449-4a94-a13e-cee4479e4d7c" containerID="a7df57889e88bac8af552482152c08d050e9b4d77f1964a72b7a64f54cabd517" exitCode=0 Oct 30 07:08:26.223003 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:26.222982 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" Oct 30 07:08:26.223064 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:26.222992 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" event={"ID":"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c","Type":"ContainerDied","Data":"a7df57889e88bac8af552482152c08d050e9b4d77f1964a72b7a64f54cabd517"} Oct 30 07:08:26.223116 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:26.223059 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx" event={"ID":"d3c2b8f1-1449-4a94-a13e-cee4479e4d7c","Type":"ContainerDied","Data":"68448250867e3afb459ef727da6b0fca3990cb9174df6d435b2e4d4622f617df"} Oct 30 07:08:26.232524 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:26.232501 4951 scope.go:122] "RemoveContainer" containerID="561e579cf71b5a17431fe29a5e60d7840ff2983f3000e5053bf4b02b754e4f4d" Oct 30 07:08:26.232873 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:08:26.232850 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"561e579cf71b5a17431fe29a5e60d7840ff2983f3000e5053bf4b02b754e4f4d\": container with ID starting with 561e579cf71b5a17431fe29a5e60d7840ff2983f3000e5053bf4b02b754e4f4d not found: ID does not exist" containerID="561e579cf71b5a17431fe29a5e60d7840ff2983f3000e5053bf4b02b754e4f4d" Oct 30 07:08:26.232971 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:26.232883 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"561e579cf71b5a17431fe29a5e60d7840ff2983f3000e5053bf4b02b754e4f4d"} err="failed to get container status \"561e579cf71b5a17431fe29a5e60d7840ff2983f3000e5053bf4b02b754e4f4d\": rpc error: code = NotFound desc = could not find container \"561e579cf71b5a17431fe29a5e60d7840ff2983f3000e5053bf4b02b754e4f4d\": container with ID starting with 561e579cf71b5a17431fe29a5e60d7840ff2983f3000e5053bf4b02b754e4f4d not found: ID does not exist" Oct 30 07:08:26.232971 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:26.232903 4951 scope.go:122] "RemoveContainer" containerID="a7df57889e88bac8af552482152c08d050e9b4d77f1964a72b7a64f54cabd517" Oct 30 07:08:26.242610 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:26.242558 4951 scope.go:122] "RemoveContainer" containerID="35010d24684ad0915687dc73daa10898db59dc7ae8b822a437a254c0f83783a2" Oct 30 07:08:26.251904 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:26.251877 4951 scope.go:122] "RemoveContainer" containerID="a7df57889e88bac8af552482152c08d050e9b4d77f1964a72b7a64f54cabd517" Oct 30 07:08:26.252193 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:08:26.252170 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7df57889e88bac8af552482152c08d050e9b4d77f1964a72b7a64f54cabd517\": container with ID starting with a7df57889e88bac8af552482152c08d050e9b4d77f1964a72b7a64f54cabd517 not found: ID does not exist" containerID="a7df57889e88bac8af552482152c08d050e9b4d77f1964a72b7a64f54cabd517" Oct 30 07:08:26.252292 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:26.252208 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7df57889e88bac8af552482152c08d050e9b4d77f1964a72b7a64f54cabd517"} err="failed to get container status \"a7df57889e88bac8af552482152c08d050e9b4d77f1964a72b7a64f54cabd517\": rpc error: code = NotFound desc = could not find container \"a7df57889e88bac8af552482152c08d050e9b4d77f1964a72b7a64f54cabd517\": container with ID starting with a7df57889e88bac8af552482152c08d050e9b4d77f1964a72b7a64f54cabd517 not found: ID does not exist" Oct 30 07:08:26.252292 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:26.252234 4951 scope.go:122] "RemoveContainer" containerID="35010d24684ad0915687dc73daa10898db59dc7ae8b822a437a254c0f83783a2" Oct 30 07:08:26.252483 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:08:26.252467 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35010d24684ad0915687dc73daa10898db59dc7ae8b822a437a254c0f83783a2\": container with ID starting with 35010d24684ad0915687dc73daa10898db59dc7ae8b822a437a254c0f83783a2 not found: ID does not exist" containerID="35010d24684ad0915687dc73daa10898db59dc7ae8b822a437a254c0f83783a2" Oct 30 07:08:26.252539 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:26.252494 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35010d24684ad0915687dc73daa10898db59dc7ae8b822a437a254c0f83783a2"} err="failed to get container status \"35010d24684ad0915687dc73daa10898db59dc7ae8b822a437a254c0f83783a2\": rpc error: code = NotFound desc = could not find container \"35010d24684ad0915687dc73daa10898db59dc7ae8b822a437a254c0f83783a2\": container with ID starting with 35010d24684ad0915687dc73daa10898db59dc7ae8b822a437a254c0f83783a2 not found: ID does not exist" Oct 30 07:08:26.268214 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:26.268187 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx"] Oct 30 07:08:26.272325 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:26.272302 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-a5d88019-gw-openshift-default-56f9d7f897-5kvfx"] Oct 30 07:08:26.305216 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:26.305182 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm"] Oct 30 07:08:26.311674 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:26.311643 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-37f3d312-68f9846df8-gxzfm"] Oct 30 07:08:26.923034 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:26.923002 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq"] Oct 30 07:08:26.923495 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:26.923262 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" podUID="342054a0-c94d-48c6-8041-edccd3763e1b" containerName="istio-proxy" containerID="cri-o://8790e1ff280218789a67887928d6071aeaccbc0d89c9f88ca7b1c4480aa3f434" gracePeriod=30 Oct 30 07:08:27.077232 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:27.076050 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr"] Oct 30 07:08:27.077232 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:27.076305 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" podUID="2895d955-4a1f-45f1-94cb-251e4996c772" containerName="istio-proxy" containerID="cri-o://30cd4c23e06b227f010dabcf29989b5ccc2450b3c721286c664db7f16953e307" gracePeriod=30 Oct 30 07:08:27.771751 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:27.771720 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e44a6d7-6cfd-4fa1-ab31-caef4639842c" path="/var/lib/kubelet/pods/1e44a6d7-6cfd-4fa1-ab31-caef4639842c/volumes" Oct 30 07:08:27.772197 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:27.772183 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3c2b8f1-1449-4a94-a13e-cee4479e4d7c" path="/var/lib/kubelet/pods/d3c2b8f1-1449-4a94-a13e-cee4479e4d7c/volumes" Oct 30 07:08:28.239703 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.239629 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72"] Oct 30 07:08:28.240114 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.240100 4951 cpu_manager.go:414] "containerMap: removing container" podUID="1e44a6d7-6cfd-4fa1-ab31-caef4639842c" containerName="maas-api" Oct 30 07:08:28.240152 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.240116 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e44a6d7-6cfd-4fa1-ab31-caef4639842c" containerName="maas-api" Oct 30 07:08:28.240152 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.240132 4951 cpu_manager.go:414] "containerMap: removing container" podUID="ee7d77ec-eae3-4640-8688-511efda613e4" containerName="maas-api" Oct 30 07:08:28.240152 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.240137 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee7d77ec-eae3-4640-8688-511efda613e4" containerName="maas-api" Oct 30 07:08:28.240238 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.240157 4951 cpu_manager.go:414] "containerMap: removing container" podUID="d3c2b8f1-1449-4a94-a13e-cee4479e4d7c" containerName="istio-proxy" Oct 30 07:08:28.240238 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.240162 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3c2b8f1-1449-4a94-a13e-cee4479e4d7c" containerName="istio-proxy" Oct 30 07:08:28.240305 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.240260 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee7d77ec-eae3-4640-8688-511efda613e4" containerName="maas-api" Oct 30 07:08:28.240305 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.240268 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3c2b8f1-1449-4a94-a13e-cee4479e4d7c" containerName="istio-proxy" Oct 30 07:08:28.240305 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.240279 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e44a6d7-6cfd-4fa1-ab31-caef4639842c" containerName="maas-api" Oct 30 07:08:28.244938 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.244906 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:28.275704 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.275678 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-b72d79-openshift-default-dockercfg-m5v5m\"" Oct 30 07:08:28.278942 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.278914 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72"] Oct 30 07:08:28.315727 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.308984 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istio-token\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:28.315727 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.309048 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-credential-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:28.315727 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.309089 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istio-data\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:28.315727 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.309123 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-workload-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:28.315727 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.309187 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istiod-ca-cert\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:28.315727 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.309219 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istio-envoy\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:28.315727 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.309244 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w96pb\" (UniqueName: \"kubernetes.io/projected/42ea0497-3b5f-4c33-a53a-bd636a4379ba-kube-api-access-w96pb\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:28.315727 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.309270 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istio-podinfo\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:28.315727 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.309314 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-workload-certs\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:28.427600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.410801 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istiod-ca-cert\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:28.427600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.410987 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istio-envoy\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:28.427600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.411029 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w96pb\" (UniqueName: \"kubernetes.io/projected/42ea0497-3b5f-4c33-a53a-bd636a4379ba-kube-api-access-w96pb\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:28.427600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.411060 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istio-podinfo\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:28.427600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.411110 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-workload-certs\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:28.427600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.411220 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istio-token\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:28.427600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.411266 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-credential-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:28.427600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.411307 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istio-data\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:28.427600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.411354 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-workload-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:28.427600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.411728 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-workload-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:28.427600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.411738 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istiod-ca-cert\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:28.427600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.412034 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-workload-certs\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:28.427600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.412298 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-credential-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:28.427600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.412688 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istio-data\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:28.430826 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.428535 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istio-podinfo\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:28.445606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.444553 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istio-envoy\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:28.454679 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.454594 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w96pb\" (UniqueName: \"kubernetes.io/projected/42ea0497-3b5f-4c33-a53a-bd636a4379ba-kube-api-access-w96pb\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:28.456957 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.456610 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istio-token\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:28.569242 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.562482 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:28.903251 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.902606 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72"] Oct 30 07:08:28.908752 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:08:28.908715 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ea0497_3b5f_4c33_a53a_bd636a4379ba.slice/crio-dac69c03db8173fd5b4fc35bc8d0f73af287a329bc606d54f688c1576f944f5a WatchSource:0}: Error finding container dac69c03db8173fd5b4fc35bc8d0f73af287a329bc606d54f688c1576f944f5a: Status 404 returned error can't find the container with id dac69c03db8173fd5b4fc35bc8d0f73af287a329bc606d54f688c1576f944f5a Oct 30 07:08:28.911389 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.911352 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:08:28.911505 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.911439 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:08:28.911566 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:28.911503 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:08:29.243905 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:29.243815 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" event={"ID":"42ea0497-3b5f-4c33-a53a-bd636a4379ba","Type":"ContainerStarted","Data":"17f52bcda66d0ed800ec1c3ef785437e87ec0ec6dae43053a48e1cbce9f9de28"} Oct 30 07:08:29.243905 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:29.243873 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" event={"ID":"42ea0497-3b5f-4c33-a53a-bd636a4379ba","Type":"ContainerStarted","Data":"dac69c03db8173fd5b4fc35bc8d0f73af287a329bc606d54f688c1576f944f5a"} Oct 30 07:08:29.287721 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:29.287665 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" podStartSLOduration=1.287648529 podStartE2EDuration="1.287648529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:08:29.286459601 +0000 UTC m=+715.629507567" watchObservedRunningTime="2026-10-30 07:08:29.287648529 +0000 UTC m=+715.630696494" Oct 30 07:08:29.563523 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:29.563483 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:29.569078 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:29.569048 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:30.250953 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:30.250924 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:30.251880 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:30.251857 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:30.624607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:30.624510 4951 patch_prober.go:29] interesting pod/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:08:30.624753 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:30.624602 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" podUID="8e2d9f9d-3fe2-4abe-8136-f34cdf8de323" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:08:31.094295 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.094259 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc"] Oct 30 07:08:31.094495 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.094458 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc" podUID="6a3b7d91-481b-453f-895e-0bd0a1799035" containerName="maas-api" containerID="cri-o://680150a1aab59880841e9f9ecafaaca569e7707d4a1831e9e5c9038f75114a7b" gracePeriod=30 Oct 30 07:08:31.258948 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.258902 4951 generic.go:356] "Generic (PLEG): container finished" podID="6a3b7d91-481b-453f-895e-0bd0a1799035" containerID="680150a1aab59880841e9f9ecafaaca569e7707d4a1831e9e5c9038f75114a7b" exitCode=0 Oct 30 07:08:31.259806 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.259776 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc" event={"ID":"6a3b7d91-481b-453f-895e-0bd0a1799035","Type":"ContainerDied","Data":"680150a1aab59880841e9f9ecafaaca569e7707d4a1831e9e5c9038f75114a7b"} Oct 30 07:08:31.262515 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.262484 4951 generic.go:356] "Generic (PLEG): container finished" podID="8e2d9f9d-3fe2-4abe-8136-f34cdf8de323" containerID="b63861a388d8570d3f07e13ffb9b28d092d4a148c81e4cd1038c48b8b3373d1c" exitCode=0 Oct 30 07:08:31.263386 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.263346 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" event={"ID":"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323","Type":"ContainerDied","Data":"b63861a388d8570d3f07e13ffb9b28d092d4a148c81e4cd1038c48b8b3373d1c"} Oct 30 07:08:31.263520 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.263414 4951 scope.go:122] "RemoveContainer" containerID="b6c985b661a41ddfcfffdb4522a7817ef37efc8f762b02ca841189f23d6108cf" Oct 30 07:08:31.413021 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.412987 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:08:31.452979 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.452950 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc" Oct 30 07:08:31.571400 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.571314 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-kube-api-access-kztg5\" (UniqueName: \"kubernetes.io/projected/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-kube-api-access-kztg5\") pod \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " Oct 30 07:08:31.571400 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.571351 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-credential-socket\") pod \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " Oct 30 07:08:31.571400 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.571379 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a3b7d91-481b-453f-895e-0bd0a1799035-kube-api-access-c4kfx\" (UniqueName: \"kubernetes.io/projected/6a3b7d91-481b-453f-895e-0bd0a1799035-kube-api-access-c4kfx\") pod \"6a3b7d91-481b-453f-895e-0bd0a1799035\" (UID: \"6a3b7d91-481b-453f-895e-0bd0a1799035\") " Oct 30 07:08:31.571727 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.571407 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istio-podinfo\") pod \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " Oct 30 07:08:31.571727 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.571428 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istiod-ca-cert\") pod \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " Oct 30 07:08:31.571727 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.571466 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istio-token\" (UniqueName: \"kubernetes.io/projected/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istio-token\") pod \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " Oct 30 07:08:31.571727 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.571497 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-workload-socket\") pod \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " Oct 30 07:08:31.571727 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.571627 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a3b7d91-481b-453f-895e-0bd0a1799035-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a3b7d91-481b-453f-895e-0bd0a1799035-openshift-service-ca\") pod \"6a3b7d91-481b-453f-895e-0bd0a1799035\" (UID: \"6a3b7d91-481b-453f-895e-0bd0a1799035\") " Oct 30 07:08:31.571727 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.571658 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-credential-socket" pod "8e2d9f9d-3fe2-4abe-8136-f34cdf8de323" (UID: "8e2d9f9d-3fe2-4abe-8136-f34cdf8de323"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:08:31.571727 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.571667 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a3b7d91-481b-453f-895e-0bd0a1799035-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a3b7d91-481b-453f-895e-0bd0a1799035-maas-api-tls\") pod \"6a3b7d91-481b-453f-895e-0bd0a1799035\" (UID: \"6a3b7d91-481b-453f-895e-0bd0a1799035\") " Oct 30 07:08:31.572066 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.571801 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-workload-socket" pod "8e2d9f9d-3fe2-4abe-8136-f34cdf8de323" (UID: "8e2d9f9d-3fe2-4abe-8136-f34cdf8de323"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:08:31.572066 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.571830 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a3b7d91-481b-453f-895e-0bd0a1799035-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a3b7d91-481b-453f-895e-0bd0a1799035-metrics-certs\") pod \"6a3b7d91-481b-453f-895e-0bd0a1799035\" (UID: \"6a3b7d91-481b-453f-895e-0bd0a1799035\") " Oct 30 07:08:31.572066 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.571899 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istiod-ca-cert" pod "8e2d9f9d-3fe2-4abe-8136-f34cdf8de323" (UID: "8e2d9f9d-3fe2-4abe-8136-f34cdf8de323"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:08:31.572066 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.571914 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istio-data\") pod \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " Oct 30 07:08:31.572066 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.571978 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istio-envoy\") pod \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " Oct 30 07:08:31.572066 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.572017 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-workload-certs\") pod \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\" (UID: \"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323\") " Oct 30 07:08:31.572382 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.572105 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a3b7d91-481b-453f-895e-0bd0a1799035-openshift-service-ca" pod "6a3b7d91-481b-453f-895e-0bd0a1799035" (UID: "6a3b7d91-481b-453f-895e-0bd0a1799035"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:08:31.572382 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.572341 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-workload-certs" pod "8e2d9f9d-3fe2-4abe-8136-f34cdf8de323" (UID: "8e2d9f9d-3fe2-4abe-8136-f34cdf8de323"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:08:31.572560 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.572538 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istio-data" pod "8e2d9f9d-3fe2-4abe-8136-f34cdf8de323" (UID: "8e2d9f9d-3fe2-4abe-8136-f34cdf8de323"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:08:31.572560 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.572551 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:31.572706 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.572599 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:31.572706 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.572616 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:31.572706 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.572630 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:31.572706 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.572644 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a3b7d91-481b-453f-895e-0bd0a1799035-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:31.574084 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.574060 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-kube-api-access-kztg5" pod "8e2d9f9d-3fe2-4abe-8136-f34cdf8de323" (UID: "8e2d9f9d-3fe2-4abe-8136-f34cdf8de323"). InnerVolumeSpecName "kube-api-access-kztg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:08:31.574179 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.574134 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istio-podinfo" pod "8e2d9f9d-3fe2-4abe-8136-f34cdf8de323" (UID: "8e2d9f9d-3fe2-4abe-8136-f34cdf8de323"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:08:31.574231 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.574193 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a3b7d91-481b-453f-895e-0bd0a1799035-metrics-certs" pod "6a3b7d91-481b-453f-895e-0bd0a1799035" (UID: "6a3b7d91-481b-453f-895e-0bd0a1799035"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:08:31.574231 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.574212 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a3b7d91-481b-453f-895e-0bd0a1799035-kube-api-access-c4kfx" pod "6a3b7d91-481b-453f-895e-0bd0a1799035" (UID: "6a3b7d91-481b-453f-895e-0bd0a1799035"). InnerVolumeSpecName "kube-api-access-c4kfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:08:31.574645 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.574624 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istio-token" pod "8e2d9f9d-3fe2-4abe-8136-f34cdf8de323" (UID: "8e2d9f9d-3fe2-4abe-8136-f34cdf8de323"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:08:31.574721 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.574669 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a3b7d91-481b-453f-895e-0bd0a1799035-maas-api-tls" pod "6a3b7d91-481b-453f-895e-0bd0a1799035" (UID: "6a3b7d91-481b-453f-895e-0bd0a1799035"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:08:31.574944 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.574927 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istio-envoy" pod "8e2d9f9d-3fe2-4abe-8136-f34cdf8de323" (UID: "8e2d9f9d-3fe2-4abe-8136-f34cdf8de323"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:08:31.674034 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.673999 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a3b7d91-481b-453f-895e-0bd0a1799035-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:31.674034 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.674031 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a3b7d91-481b-453f-895e-0bd0a1799035-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:31.674034 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.674040 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:31.674266 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.674048 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:31.674266 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.674057 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kztg5\" (UniqueName: \"kubernetes.io/projected/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-kube-api-access-kztg5\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:31.674266 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.674066 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4kfx\" (UniqueName: \"kubernetes.io/projected/6a3b7d91-481b-453f-895e-0bd0a1799035-kube-api-access-c4kfx\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:31.674266 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.674076 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:31.674266 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:31.674086 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:32.238316 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.235852 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:08:32.287611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.284382 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc" event={"ID":"6a3b7d91-481b-453f-895e-0bd0a1799035","Type":"ContainerDied","Data":"074273167b4183d2ee7c1813de90f1d238892ee5396add37454c2ed4abec16c3"} Oct 30 07:08:32.287611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.284435 4951 scope.go:122] "RemoveContainer" containerID="680150a1aab59880841e9f9ecafaaca569e7707d4a1831e9e5c9038f75114a7b" Oct 30 07:08:32.287611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.284520 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc" Oct 30 07:08:32.288277 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.288104 4951 generic.go:356] "Generic (PLEG): container finished" podID="2895d955-4a1f-45f1-94cb-251e4996c772" containerID="30cd4c23e06b227f010dabcf29989b5ccc2450b3c721286c664db7f16953e307" exitCode=0 Oct 30 07:08:32.288277 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.288174 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" event={"ID":"2895d955-4a1f-45f1-94cb-251e4996c772","Type":"ContainerDied","Data":"30cd4c23e06b227f010dabcf29989b5ccc2450b3c721286c664db7f16953e307"} Oct 30 07:08:32.295098 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.294768 4951 generic.go:356] "Generic (PLEG): container finished" podID="342054a0-c94d-48c6-8041-edccd3763e1b" containerID="8790e1ff280218789a67887928d6071aeaccbc0d89c9f88ca7b1c4480aa3f434" exitCode=0 Oct 30 07:08:32.295098 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.295001 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" event={"ID":"342054a0-c94d-48c6-8041-edccd3763e1b","Type":"ContainerDied","Data":"8790e1ff280218789a67887928d6071aeaccbc0d89c9f88ca7b1c4480aa3f434"} Oct 30 07:08:32.295300 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.295115 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" event={"ID":"342054a0-c94d-48c6-8041-edccd3763e1b","Type":"ContainerDied","Data":"1cec421bd4458f16673f99570ed70bb796f397ca7267c0b7851bc42e746a921b"} Oct 30 07:08:32.295389 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.295306 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq" Oct 30 07:08:32.306316 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.304753 4951 scope.go:122] "RemoveContainer" containerID="1556c961a76426de9668eef058d93a69e4dc81cf00f9dfb6203cd2334dd6dcb8" Oct 30 07:08:32.314013 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.312163 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" Oct 30 07:08:32.314013 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.312175 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq" event={"ID":"8e2d9f9d-3fe2-4abe-8136-f34cdf8de323","Type":"ContainerDied","Data":"b5cc6811ef408ff64fd9f6a630c2af2f0e40b2a2269e6c58178a86f48e90e724"} Oct 30 07:08:32.331388 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.331370 4951 scope.go:122] "RemoveContainer" containerID="8790e1ff280218789a67887928d6071aeaccbc0d89c9f88ca7b1c4480aa3f434" Oct 30 07:08:32.338825 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.338803 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc"] Oct 30 07:08:32.343180 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.343131 4951 scope.go:122] "RemoveContainer" containerID="f748aee0e36fbb055f22676f040610fa3a85d471a8eacb1e2b20ae379b8d5f59" Oct 30 07:08:32.351031 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.350979 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-77bbf25a-77f4f8f68-tq5nc"] Oct 30 07:08:32.353760 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.353677 4951 scope.go:122] "RemoveContainer" containerID="8790e1ff280218789a67887928d6071aeaccbc0d89c9f88ca7b1c4480aa3f434" Oct 30 07:08:32.354103 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:08:32.353973 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8790e1ff280218789a67887928d6071aeaccbc0d89c9f88ca7b1c4480aa3f434\": container with ID starting with 8790e1ff280218789a67887928d6071aeaccbc0d89c9f88ca7b1c4480aa3f434 not found: ID does not exist" containerID="8790e1ff280218789a67887928d6071aeaccbc0d89c9f88ca7b1c4480aa3f434" Oct 30 07:08:32.354103 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.354008 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8790e1ff280218789a67887928d6071aeaccbc0d89c9f88ca7b1c4480aa3f434"} err="failed to get container status \"8790e1ff280218789a67887928d6071aeaccbc0d89c9f88ca7b1c4480aa3f434\": rpc error: code = NotFound desc = could not find container \"8790e1ff280218789a67887928d6071aeaccbc0d89c9f88ca7b1c4480aa3f434\": container with ID starting with 8790e1ff280218789a67887928d6071aeaccbc0d89c9f88ca7b1c4480aa3f434 not found: ID does not exist" Oct 30 07:08:32.354103 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.354034 4951 scope.go:122] "RemoveContainer" containerID="f748aee0e36fbb055f22676f040610fa3a85d471a8eacb1e2b20ae379b8d5f59" Oct 30 07:08:32.354819 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:08:32.354434 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f748aee0e36fbb055f22676f040610fa3a85d471a8eacb1e2b20ae379b8d5f59\": container with ID starting with f748aee0e36fbb055f22676f040610fa3a85d471a8eacb1e2b20ae379b8d5f59 not found: ID does not exist" containerID="f748aee0e36fbb055f22676f040610fa3a85d471a8eacb1e2b20ae379b8d5f59" Oct 30 07:08:32.354819 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.354464 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f748aee0e36fbb055f22676f040610fa3a85d471a8eacb1e2b20ae379b8d5f59"} err="failed to get container status \"f748aee0e36fbb055f22676f040610fa3a85d471a8eacb1e2b20ae379b8d5f59\": rpc error: code = NotFound desc = could not find container \"f748aee0e36fbb055f22676f040610fa3a85d471a8eacb1e2b20ae379b8d5f59\": container with ID starting with f748aee0e36fbb055f22676f040610fa3a85d471a8eacb1e2b20ae379b8d5f59 not found: ID does not exist" Oct 30 07:08:32.354819 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.354486 4951 scope.go:122] "RemoveContainer" containerID="b63861a388d8570d3f07e13ffb9b28d092d4a148c81e4cd1038c48b8b3373d1c" Oct 30 07:08:32.383874 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.383849 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-workload-certs\") pod \"342054a0-c94d-48c6-8041-edccd3763e1b\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " Oct 30 07:08:32.383983 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.383891 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/342054a0-c94d-48c6-8041-edccd3763e1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342054a0-c94d-48c6-8041-edccd3763e1b-istiod-ca-cert\") pod \"342054a0-c94d-48c6-8041-edccd3763e1b\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " Oct 30 07:08:32.383983 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.383942 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/342054a0-c94d-48c6-8041-edccd3763e1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342054a0-c94d-48c6-8041-edccd3763e1b-istio-podinfo\") pod \"342054a0-c94d-48c6-8041-edccd3763e1b\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " Oct 30 07:08:32.383983 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.383973 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-istio-data\") pod \"342054a0-c94d-48c6-8041-edccd3763e1b\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " Oct 30 07:08:32.384102 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.384040 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/342054a0-c94d-48c6-8041-edccd3763e1b-istio-token\" (UniqueName: \"kubernetes.io/projected/342054a0-c94d-48c6-8041-edccd3763e1b-istio-token\") pod \"342054a0-c94d-48c6-8041-edccd3763e1b\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " Oct 30 07:08:32.384102 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.384065 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/342054a0-c94d-48c6-8041-edccd3763e1b-kube-api-access-9dpcx\" (UniqueName: \"kubernetes.io/projected/342054a0-c94d-48c6-8041-edccd3763e1b-kube-api-access-9dpcx\") pod \"342054a0-c94d-48c6-8041-edccd3763e1b\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " Oct 30 07:08:32.384161 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.384095 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-workload-certs" pod "342054a0-c94d-48c6-8041-edccd3763e1b" (UID: "342054a0-c94d-48c6-8041-edccd3763e1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:08:32.384161 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.384102 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-workload-socket\") pod \"342054a0-c94d-48c6-8041-edccd3763e1b\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " Oct 30 07:08:32.384229 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.384199 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-istio-envoy\") pod \"342054a0-c94d-48c6-8041-edccd3763e1b\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " Oct 30 07:08:32.384270 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.384240 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-credential-socket\") pod \"342054a0-c94d-48c6-8041-edccd3763e1b\" (UID: \"342054a0-c94d-48c6-8041-edccd3763e1b\") " Oct 30 07:08:32.384316 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.384296 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-istio-data" pod "342054a0-c94d-48c6-8041-edccd3763e1b" (UID: "342054a0-c94d-48c6-8041-edccd3763e1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:08:32.384380 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.384308 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-workload-socket" pod "342054a0-c94d-48c6-8041-edccd3763e1b" (UID: "342054a0-c94d-48c6-8041-edccd3763e1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:08:32.384380 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.384344 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/342054a0-c94d-48c6-8041-edccd3763e1b-istiod-ca-cert" pod "342054a0-c94d-48c6-8041-edccd3763e1b" (UID: "342054a0-c94d-48c6-8041-edccd3763e1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:08:32.384754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.384710 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-credential-socket" pod "342054a0-c94d-48c6-8041-edccd3763e1b" (UID: "342054a0-c94d-48c6-8041-edccd3763e1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:08:32.384856 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.384756 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:32.384856 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.384776 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342054a0-c94d-48c6-8041-edccd3763e1b-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:32.384856 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.384790 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:32.384856 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.384801 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:32.387409 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.387374 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-istio-envoy" pod "342054a0-c94d-48c6-8041-edccd3763e1b" (UID: "342054a0-c94d-48c6-8041-edccd3763e1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:08:32.387409 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.387392 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/342054a0-c94d-48c6-8041-edccd3763e1b-kube-api-access-9dpcx" pod "342054a0-c94d-48c6-8041-edccd3763e1b" (UID: "342054a0-c94d-48c6-8041-edccd3763e1b"). InnerVolumeSpecName "kube-api-access-9dpcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:08:32.387528 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.387419 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/342054a0-c94d-48c6-8041-edccd3763e1b-istio-podinfo" pod "342054a0-c94d-48c6-8041-edccd3763e1b" (UID: "342054a0-c94d-48c6-8041-edccd3763e1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:08:32.387528 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.387453 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/342054a0-c94d-48c6-8041-edccd3763e1b-istio-token" pod "342054a0-c94d-48c6-8041-edccd3763e1b" (UID: "342054a0-c94d-48c6-8041-edccd3763e1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:08:32.393377 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.393293 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq"] Oct 30 07:08:32.400905 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.400883 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9bd1c95c-openshift-default-69458ccb89-llghq"] Oct 30 07:08:32.414323 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.414303 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:08:32.485335 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.485254 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-istio-data\") pod \"2895d955-4a1f-45f1-94cb-251e4996c772\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " Oct 30 07:08:32.485335 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.485286 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2895d955-4a1f-45f1-94cb-251e4996c772-istio-token\" (UniqueName: \"kubernetes.io/projected/2895d955-4a1f-45f1-94cb-251e4996c772-istio-token\") pod \"2895d955-4a1f-45f1-94cb-251e4996c772\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " Oct 30 07:08:32.485335 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.485323 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-workload-certs\") pod \"2895d955-4a1f-45f1-94cb-251e4996c772\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " Oct 30 07:08:32.485641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.485355 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2895d955-4a1f-45f1-94cb-251e4996c772-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2895d955-4a1f-45f1-94cb-251e4996c772-istiod-ca-cert\") pod \"2895d955-4a1f-45f1-94cb-251e4996c772\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " Oct 30 07:08:32.485641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.485386 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-credential-socket\") pod \"2895d955-4a1f-45f1-94cb-251e4996c772\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " Oct 30 07:08:32.485641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.485419 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-workload-socket\") pod \"2895d955-4a1f-45f1-94cb-251e4996c772\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " Oct 30 07:08:32.485641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.485479 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2895d955-4a1f-45f1-94cb-251e4996c772-kube-api-access-627tq\" (UniqueName: \"kubernetes.io/projected/2895d955-4a1f-45f1-94cb-251e4996c772-kube-api-access-627tq\") pod \"2895d955-4a1f-45f1-94cb-251e4996c772\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " Oct 30 07:08:32.485641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.485547 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2895d955-4a1f-45f1-94cb-251e4996c772-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2895d955-4a1f-45f1-94cb-251e4996c772-istio-podinfo\") pod \"2895d955-4a1f-45f1-94cb-251e4996c772\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " Oct 30 07:08:32.485641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.485564 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-istio-data" pod "2895d955-4a1f-45f1-94cb-251e4996c772" (UID: "2895d955-4a1f-45f1-94cb-251e4996c772"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:08:32.485985 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.485654 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-workload-certs" pod "2895d955-4a1f-45f1-94cb-251e4996c772" (UID: "2895d955-4a1f-45f1-94cb-251e4996c772"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:08:32.485985 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.485696 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-istio-envoy\") pod \"2895d955-4a1f-45f1-94cb-251e4996c772\" (UID: \"2895d955-4a1f-45f1-94cb-251e4996c772\") " Oct 30 07:08:32.485985 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.485826 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-credential-socket" pod "2895d955-4a1f-45f1-94cb-251e4996c772" (UID: "2895d955-4a1f-45f1-94cb-251e4996c772"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:08:32.485985 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.485842 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-workload-socket" pod "2895d955-4a1f-45f1-94cb-251e4996c772" (UID: "2895d955-4a1f-45f1-94cb-251e4996c772"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:08:32.486191 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.486131 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:32.486191 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.486154 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342054a0-c94d-48c6-8041-edccd3763e1b-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:32.486191 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.486158 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2895d955-4a1f-45f1-94cb-251e4996c772-istiod-ca-cert" pod "2895d955-4a1f-45f1-94cb-251e4996c772" (UID: "2895d955-4a1f-45f1-94cb-251e4996c772"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:08:32.486191 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.486170 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:32.486191 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.486183 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:32.486435 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.486197 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:32.486435 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.486212 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:32.486435 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.486231 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342054a0-c94d-48c6-8041-edccd3763e1b-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:32.486435 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.486247 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/342054a0-c94d-48c6-8041-edccd3763e1b-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:32.486435 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.486261 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dpcx\" (UniqueName: \"kubernetes.io/projected/342054a0-c94d-48c6-8041-edccd3763e1b-kube-api-access-9dpcx\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:32.488073 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.488042 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-istio-envoy" pod "2895d955-4a1f-45f1-94cb-251e4996c772" (UID: "2895d955-4a1f-45f1-94cb-251e4996c772"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:08:32.488186 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.488108 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2895d955-4a1f-45f1-94cb-251e4996c772-kube-api-access-627tq" pod "2895d955-4a1f-45f1-94cb-251e4996c772" (UID: "2895d955-4a1f-45f1-94cb-251e4996c772"). InnerVolumeSpecName "kube-api-access-627tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:08:32.488186 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.488132 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2895d955-4a1f-45f1-94cb-251e4996c772-istio-token" pod "2895d955-4a1f-45f1-94cb-251e4996c772" (UID: "2895d955-4a1f-45f1-94cb-251e4996c772"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:08:32.488186 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.488179 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2895d955-4a1f-45f1-94cb-251e4996c772-istio-podinfo" pod "2895d955-4a1f-45f1-94cb-251e4996c772" (UID: "2895d955-4a1f-45f1-94cb-251e4996c772"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:08:32.587916 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.587874 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2895d955-4a1f-45f1-94cb-251e4996c772-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:32.587916 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.587917 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2895d955-4a1f-45f1-94cb-251e4996c772-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:32.588152 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.587932 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2895d955-4a1f-45f1-94cb-251e4996c772-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:32.588152 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.587947 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-627tq\" (UniqueName: \"kubernetes.io/projected/2895d955-4a1f-45f1-94cb-251e4996c772-kube-api-access-627tq\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:32.588152 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.587963 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2895d955-4a1f-45f1-94cb-251e4996c772-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:08:32.641211 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.641175 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq"] Oct 30 07:08:32.655316 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:32.655281 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-37f3d312-openshift-default-5b86f69987-lr8qq"] Oct 30 07:08:33.318693 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.318613 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" Oct 30 07:08:33.319132 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.318606 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr" event={"ID":"2895d955-4a1f-45f1-94cb-251e4996c772","Type":"ContainerDied","Data":"7b9eeb9a2fe0b25811586d9a5dd7b63d40f2a39933010af0bc6c6455f7bd7eeb"} Oct 30 07:08:33.319132 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.318761 4951 scope.go:122] "RemoveContainer" containerID="30cd4c23e06b227f010dabcf29989b5ccc2450b3c721286c664db7f16953e307" Oct 30 07:08:33.374920 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.374891 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-2f845b-5d5d94cf95-jsspt"] Oct 30 07:08:33.375461 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.375440 4951 cpu_manager.go:414] "containerMap: removing container" podUID="8e2d9f9d-3fe2-4abe-8136-f34cdf8de323" containerName="istio-proxy" Oct 30 07:08:33.375461 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.375461 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e2d9f9d-3fe2-4abe-8136-f34cdf8de323" containerName="istio-proxy" Oct 30 07:08:33.375676 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.375478 4951 cpu_manager.go:414] "containerMap: removing container" podUID="8e2d9f9d-3fe2-4abe-8136-f34cdf8de323" containerName="istio-proxy" Oct 30 07:08:33.375676 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.375484 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e2d9f9d-3fe2-4abe-8136-f34cdf8de323" containerName="istio-proxy" Oct 30 07:08:33.375676 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.375490 4951 cpu_manager.go:414] "containerMap: removing container" podUID="6a3b7d91-481b-453f-895e-0bd0a1799035" containerName="maas-api" Oct 30 07:08:33.375676 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.375495 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a3b7d91-481b-453f-895e-0bd0a1799035" containerName="maas-api" Oct 30 07:08:33.375676 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.375504 4951 cpu_manager.go:414] "containerMap: removing container" podUID="342054a0-c94d-48c6-8041-edccd3763e1b" containerName="istio-proxy" Oct 30 07:08:33.375676 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.375510 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="342054a0-c94d-48c6-8041-edccd3763e1b" containerName="istio-proxy" Oct 30 07:08:33.375676 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.375520 4951 cpu_manager.go:414] "containerMap: removing container" podUID="2895d955-4a1f-45f1-94cb-251e4996c772" containerName="istio-proxy" Oct 30 07:08:33.375676 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.375528 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2895d955-4a1f-45f1-94cb-251e4996c772" containerName="istio-proxy" Oct 30 07:08:33.375676 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.375538 4951 cpu_manager.go:414] "containerMap: removing container" podUID="2895d955-4a1f-45f1-94cb-251e4996c772" containerName="istio-proxy" Oct 30 07:08:33.375676 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.375546 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2895d955-4a1f-45f1-94cb-251e4996c772" containerName="istio-proxy" Oct 30 07:08:33.375676 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.375559 4951 cpu_manager.go:414] "containerMap: removing container" podUID="d3c2b8f1-1449-4a94-a13e-cee4479e4d7c" containerName="istio-proxy" Oct 30 07:08:33.375676 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.375563 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3c2b8f1-1449-4a94-a13e-cee4479e4d7c" containerName="istio-proxy" Oct 30 07:08:33.375676 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.375622 4951 cpu_manager.go:414] "containerMap: removing container" podUID="342054a0-c94d-48c6-8041-edccd3763e1b" containerName="istio-proxy" Oct 30 07:08:33.375676 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.375628 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="342054a0-c94d-48c6-8041-edccd3763e1b" containerName="istio-proxy" Oct 30 07:08:33.376225 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.375729 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="342054a0-c94d-48c6-8041-edccd3763e1b" containerName="istio-proxy" Oct 30 07:08:33.376225 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.375745 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e2d9f9d-3fe2-4abe-8136-f34cdf8de323" containerName="istio-proxy" Oct 30 07:08:33.376225 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.375755 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e2d9f9d-3fe2-4abe-8136-f34cdf8de323" containerName="istio-proxy" Oct 30 07:08:33.376225 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.375763 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a3b7d91-481b-453f-895e-0bd0a1799035" containerName="maas-api" Oct 30 07:08:33.376225 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.375774 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="2895d955-4a1f-45f1-94cb-251e4996c772" containerName="istio-proxy" Oct 30 07:08:33.376225 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.375782 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="2895d955-4a1f-45f1-94cb-251e4996c772" containerName="istio-proxy" Oct 30 07:08:33.376225 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.375790 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3c2b8f1-1449-4a94-a13e-cee4479e4d7c" containerName="istio-proxy" Oct 30 07:08:33.380453 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.380435 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2f845b-5d5d94cf95-jsspt" Oct 30 07:08:33.399069 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.399044 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-2f845b\"" Oct 30 07:08:33.403260 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.403232 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr"] Oct 30 07:08:33.409942 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.409919 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-2f845b-5d5d94cf95-jsspt"] Oct 30 07:08:33.412894 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.412871 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7a77072b-openshift-default-67bfcb4f8c-ngrwr"] Oct 30 07:08:33.461971 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.461933 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2f845b-7d977fd55d-5ctj5"] Oct 30 07:08:33.462429 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.462414 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="342054a0-c94d-48c6-8041-edccd3763e1b" containerName="istio-proxy" Oct 30 07:08:33.465606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.465586 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-2f845b-7d977fd55d-5ctj5" Oct 30 07:08:33.479461 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.479428 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-2f845b-dockercfg-tchmf\"" Oct 30 07:08:33.493510 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.493473 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2f845b-7d977fd55d-5ctj5"] Oct 30 07:08:33.497009 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.496978 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp6vt\" (UniqueName: \"kubernetes.io/projected/4b05487e-0050-4d5a-8875-4878d2b565a4-kube-api-access-zp6vt\") pod \"payload-pre-processing-e2e-worker-w3-2f845b-5d5d94cf95-jsspt\" (UID: \"4b05487e-0050-4d5a-8875-4878d2b565a4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2f845b-5d5d94cf95-jsspt" Oct 30 07:08:33.497157 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.497041 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b05487e-0050-4d5a-8875-4878d2b565a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-2f845b-5d5d94cf95-jsspt\" (UID: \"4b05487e-0050-4d5a-8875-4878d2b565a4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2f845b-5d5d94cf95-jsspt" Oct 30 07:08:33.539393 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.539333 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr"] Oct 30 07:08:33.543226 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.543205 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:08:33.568235 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.568208 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a0b3f5-openshift-default-dockercfg-kzcpc\"" Oct 30 07:08:33.579291 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.579224 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr"] Oct 30 07:08:33.598390 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.598355 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfhkn\" (UniqueName: \"kubernetes.io/projected/798d3036-be81-460a-819d-2385a159ecb1-kube-api-access-wfhkn\") pod \"payload-processing-e2e-worker-w3-2f845b-7d977fd55d-5ctj5\" (UID: \"798d3036-be81-460a-819d-2385a159ecb1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2f845b-7d977fd55d-5ctj5" Oct 30 07:08:33.598544 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.598437 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp6vt\" (UniqueName: \"kubernetes.io/projected/4b05487e-0050-4d5a-8875-4878d2b565a4-kube-api-access-zp6vt\") pod \"payload-pre-processing-e2e-worker-w3-2f845b-5d5d94cf95-jsspt\" (UID: \"4b05487e-0050-4d5a-8875-4878d2b565a4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2f845b-5d5d94cf95-jsspt" Oct 30 07:08:33.598544 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.598478 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/798d3036-be81-460a-819d-2385a159ecb1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-2f845b-7d977fd55d-5ctj5\" (UID: \"798d3036-be81-460a-819d-2385a159ecb1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2f845b-7d977fd55d-5ctj5" Oct 30 07:08:33.598544 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.598522 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b05487e-0050-4d5a-8875-4878d2b565a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-2f845b-5d5d94cf95-jsspt\" (UID: \"4b05487e-0050-4d5a-8875-4878d2b565a4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2f845b-5d5d94cf95-jsspt" Oct 30 07:08:33.599713 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.599685 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b05487e-0050-4d5a-8875-4878d2b565a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-2f845b-5d5d94cf95-jsspt\" (UID: \"4b05487e-0050-4d5a-8875-4878d2b565a4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2f845b-5d5d94cf95-jsspt" Oct 30 07:08:33.624542 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.624496 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp6vt\" (UniqueName: \"kubernetes.io/projected/4b05487e-0050-4d5a-8875-4878d2b565a4-kube-api-access-zp6vt\") pod \"payload-pre-processing-e2e-worker-w3-2f845b-5d5d94cf95-jsspt\" (UID: \"4b05487e-0050-4d5a-8875-4878d2b565a4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2f845b-5d5d94cf95-jsspt" Oct 30 07:08:33.702607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.701091 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:08:33.702607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.701195 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:08:33.702607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.701261 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:08:33.702607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.701293 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:08:33.702607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.701345 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfhkn\" (UniqueName: \"kubernetes.io/projected/798d3036-be81-460a-819d-2385a159ecb1-kube-api-access-wfhkn\") pod \"payload-processing-e2e-worker-w3-2f845b-7d977fd55d-5ctj5\" (UID: \"798d3036-be81-460a-819d-2385a159ecb1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2f845b-7d977fd55d-5ctj5" Oct 30 07:08:33.702607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.701378 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:08:33.702607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.701423 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:08:33.702607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.701470 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:08:33.702607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.701501 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/798d3036-be81-460a-819d-2385a159ecb1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-2f845b-7d977fd55d-5ctj5\" (UID: \"798d3036-be81-460a-819d-2385a159ecb1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2f845b-7d977fd55d-5ctj5" Oct 30 07:08:33.702607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.701552 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdbph\" (UniqueName: \"kubernetes.io/projected/de840f64-9d0c-4e5b-aa38-19c054fc8b17-kube-api-access-wdbph\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:08:33.702607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.701597 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:08:33.703307 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.703007 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/798d3036-be81-460a-819d-2385a159ecb1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-2f845b-7d977fd55d-5ctj5\" (UID: \"798d3036-be81-460a-819d-2385a159ecb1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2f845b-7d977fd55d-5ctj5" Oct 30 07:08:33.713082 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.711764 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2f845b-5d5d94cf95-jsspt" Oct 30 07:08:33.726948 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.725107 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfhkn\" (UniqueName: \"kubernetes.io/projected/798d3036-be81-460a-819d-2385a159ecb1-kube-api-access-wfhkn\") pod \"payload-processing-e2e-worker-w3-2f845b-7d977fd55d-5ctj5\" (UID: \"798d3036-be81-460a-819d-2385a159ecb1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2f845b-7d977fd55d-5ctj5" Oct 30 07:08:33.788495 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.786110 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2895d955-4a1f-45f1-94cb-251e4996c772" path="/var/lib/kubelet/pods/2895d955-4a1f-45f1-94cb-251e4996c772/volumes" Oct 30 07:08:33.788495 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.786740 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="342054a0-c94d-48c6-8041-edccd3763e1b" path="/var/lib/kubelet/pods/342054a0-c94d-48c6-8041-edccd3763e1b/volumes" Oct 30 07:08:33.788495 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.787251 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a3b7d91-481b-453f-895e-0bd0a1799035" path="/var/lib/kubelet/pods/6a3b7d91-481b-453f-895e-0bd0a1799035/volumes" Oct 30 07:08:33.788495 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.788227 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e2d9f9d-3fe2-4abe-8136-f34cdf8de323" path="/var/lib/kubelet/pods/8e2d9f9d-3fe2-4abe-8136-f34cdf8de323/volumes" Oct 30 07:08:33.797622 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.796529 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-2f845b-dockercfg-tchmf\"" Oct 30 07:08:33.803167 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.803139 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:08:33.803291 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.803188 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:08:33.803291 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.803210 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:08:33.803291 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.803237 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:08:33.803291 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.803264 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:08:33.803527 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.803293 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:08:33.803527 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.803325 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdbph\" (UniqueName: \"kubernetes.io/projected/de840f64-9d0c-4e5b-aa38-19c054fc8b17-kube-api-access-wdbph\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:08:33.803527 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.803343 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:08:33.803527 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.803368 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:08:33.803795 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.803733 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:08:33.803908 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.803887 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:08:33.803974 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.803959 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-2f845b-7d977fd55d-5ctj5" Oct 30 07:08:33.804276 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.804256 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:08:33.805025 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.804998 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:08:33.805529 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.805482 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:08:33.807538 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.807515 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:08:33.808222 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.807969 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:08:33.828048 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.828017 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdbph\" (UniqueName: \"kubernetes.io/projected/de840f64-9d0c-4e5b-aa38-19c054fc8b17-kube-api-access-wdbph\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:08:33.829168 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.829120 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:08:33.873225 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.873197 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a0b3f5-openshift-default-dockercfg-kzcpc\"" Oct 30 07:08:33.881123 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:33.881036 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:08:34.043790 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:34.043759 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-2f845b-5d5d94cf95-jsspt"] Oct 30 07:08:34.118646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:34.118617 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2f845b-7d977fd55d-5ctj5"] Oct 30 07:08:34.124760 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:08:34.124720 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod798d3036_be81_460a_819d_2385a159ecb1.slice/crio-5a3b94bd12be78c4b315284e1cbccd72eba5d89fc85c615b92d7cd07b1b33db2 WatchSource:0}: Error finding container 5a3b94bd12be78c4b315284e1cbccd72eba5d89fc85c615b92d7cd07b1b33db2: Status 404 returned error can't find the container with id 5a3b94bd12be78c4b315284e1cbccd72eba5d89fc85c615b92d7cd07b1b33db2 Oct 30 07:08:34.189927 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:34.189906 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr"] Oct 30 07:08:34.192056 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:08:34.192033 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde840f64_9d0c_4e5b_aa38_19c054fc8b17.slice/crio-7d4a5d33096adce750b56bbee136e3d55ef543e1941ebf87acf479d35d714502 WatchSource:0}: Error finding container 7d4a5d33096adce750b56bbee136e3d55ef543e1941ebf87acf479d35d714502: Status 404 returned error can't find the container with id 7d4a5d33096adce750b56bbee136e3d55ef543e1941ebf87acf479d35d714502 Oct 30 07:08:34.194267 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:34.194236 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:08:34.194348 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:34.194298 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:08:34.194348 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:34.194331 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:08:34.327257 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:34.327216 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" event={"ID":"de840f64-9d0c-4e5b-aa38-19c054fc8b17","Type":"ContainerStarted","Data":"b71ecc855f084d3301e8e09f292911517600b11035063e209cff52d778f237c0"} Oct 30 07:08:34.327822 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:34.327792 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" event={"ID":"de840f64-9d0c-4e5b-aa38-19c054fc8b17","Type":"ContainerStarted","Data":"7d4a5d33096adce750b56bbee136e3d55ef543e1941ebf87acf479d35d714502"} Oct 30 07:08:34.328960 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:34.328933 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-2f845b-7d977fd55d-5ctj5" event={"ID":"798d3036-be81-460a-819d-2385a159ecb1","Type":"ContainerStarted","Data":"ef8f638005f7a4c7849ac2edd2c19c73b9bb584d829cef3a9547fbadba632bbe"} Oct 30 07:08:34.329048 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:34.328982 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-2f845b-7d977fd55d-5ctj5" event={"ID":"798d3036-be81-460a-819d-2385a159ecb1","Type":"ContainerStarted","Data":"5a3b94bd12be78c4b315284e1cbccd72eba5d89fc85c615b92d7cd07b1b33db2"} Oct 30 07:08:34.329109 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:34.329056 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-2f845b-7d977fd55d-5ctj5" Oct 30 07:08:34.330552 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:34.330529 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2f845b-5d5d94cf95-jsspt" event={"ID":"4b05487e-0050-4d5a-8875-4878d2b565a4","Type":"ContainerStarted","Data":"30dc748d281ff7f18235b353bc665df4684dc71ffbd616a6590039b84470d454"} Oct 30 07:08:34.330636 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:34.330606 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2f845b-5d5d94cf95-jsspt" event={"ID":"4b05487e-0050-4d5a-8875-4878d2b565a4","Type":"ContainerStarted","Data":"90f2ce31b9400c98951c64bc3b8b8e11497c7b79ee1ab99e72ce6a7cabb2247d"} Oct 30 07:08:34.330689 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:34.330641 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2f845b-5d5d94cf95-jsspt" Oct 30 07:08:34.372596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:34.372493 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" podStartSLOduration=1.372474549 podStartE2EDuration="1.372474549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:08:34.369429713 +0000 UTC m=+720.712477696" watchObservedRunningTime="2026-10-30 07:08:34.372474549 +0000 UTC m=+720.715522514" Oct 30 07:08:34.408866 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:34.408805 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-2f845b-7d977fd55d-5ctj5" podStartSLOduration=1.408784747 podStartE2EDuration="1.408784747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:08:34.406869448 +0000 UTC m=+720.749917446" watchObservedRunningTime="2026-10-30 07:08:34.408784747 +0000 UTC m=+720.751832767" Oct 30 07:08:34.446177 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:34.446118 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2f845b-5d5d94cf95-jsspt" podStartSLOduration=1.446097941 podStartE2EDuration="1.446097941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:08:34.444980264 +0000 UTC m=+720.788028230" watchObservedRunningTime="2026-10-30 07:08:34.446097941 +0000 UTC m=+720.789145904" Oct 30 07:08:34.881394 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:34.881354 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:08:35.206029 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.205948 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b72d79-7fd54fd9d4-hfflc"] Oct 30 07:08:35.209220 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.209199 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b72d79-7fd54fd9d4-hfflc" Oct 30 07:08:35.227919 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.227890 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-b72d79\"" Oct 30 07:08:35.228160 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.228139 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-b72d79-dockercfg-wm5px\"" Oct 30 07:08:35.230533 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.230513 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b72d79-7fd54fd9d4-hfflc"] Oct 30 07:08:35.262864 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.262836 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b72d79-77d45dc4f5-nzd72"] Oct 30 07:08:35.266434 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.266412 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b72d79-77d45dc4f5-nzd72" Oct 30 07:08:35.289148 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.289107 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b72d79-77d45dc4f5-nzd72"] Oct 30 07:08:35.318826 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.317656 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvrb2\" (UniqueName: \"kubernetes.io/projected/cd68fbab-46a8-4bec-8c75-e6d3e07bf691-kube-api-access-rvrb2\") pod \"payload-processing-e2e-worker-w0-b72d79-7fd54fd9d4-hfflc\" (UID: \"cd68fbab-46a8-4bec-8c75-e6d3e07bf691\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b72d79-7fd54fd9d4-hfflc" Oct 30 07:08:35.318826 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.317779 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd68fbab-46a8-4bec-8c75-e6d3e07bf691-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b72d79-7fd54fd9d4-hfflc\" (UID: \"cd68fbab-46a8-4bec-8c75-e6d3e07bf691\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b72d79-7fd54fd9d4-hfflc" Oct 30 07:08:35.418654 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.418618 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50da1f49-e013-4e3e-a139-99d4ae6d668f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b72d79-77d45dc4f5-nzd72\" (UID: \"50da1f49-e013-4e3e-a139-99d4ae6d668f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b72d79-77d45dc4f5-nzd72" Oct 30 07:08:35.419187 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.418692 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvrb2\" (UniqueName: \"kubernetes.io/projected/cd68fbab-46a8-4bec-8c75-e6d3e07bf691-kube-api-access-rvrb2\") pod \"payload-processing-e2e-worker-w0-b72d79-7fd54fd9d4-hfflc\" (UID: \"cd68fbab-46a8-4bec-8c75-e6d3e07bf691\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b72d79-7fd54fd9d4-hfflc" Oct 30 07:08:35.419187 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.418818 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djnnw\" (UniqueName: \"kubernetes.io/projected/50da1f49-e013-4e3e-a139-99d4ae6d668f-kube-api-access-djnnw\") pod \"payload-pre-processing-e2e-worker-w0-b72d79-77d45dc4f5-nzd72\" (UID: \"50da1f49-e013-4e3e-a139-99d4ae6d668f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b72d79-77d45dc4f5-nzd72" Oct 30 07:08:35.419187 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.418868 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd68fbab-46a8-4bec-8c75-e6d3e07bf691-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b72d79-7fd54fd9d4-hfflc\" (UID: \"cd68fbab-46a8-4bec-8c75-e6d3e07bf691\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b72d79-7fd54fd9d4-hfflc" Oct 30 07:08:35.419641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.419621 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd68fbab-46a8-4bec-8c75-e6d3e07bf691-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b72d79-7fd54fd9d4-hfflc\" (UID: \"cd68fbab-46a8-4bec-8c75-e6d3e07bf691\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b72d79-7fd54fd9d4-hfflc" Oct 30 07:08:35.441327 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.441298 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvrb2\" (UniqueName: \"kubernetes.io/projected/cd68fbab-46a8-4bec-8c75-e6d3e07bf691-kube-api-access-rvrb2\") pod \"payload-processing-e2e-worker-w0-b72d79-7fd54fd9d4-hfflc\" (UID: \"cd68fbab-46a8-4bec-8c75-e6d3e07bf691\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b72d79-7fd54fd9d4-hfflc" Oct 30 07:08:35.520153 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.520120 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djnnw\" (UniqueName: \"kubernetes.io/projected/50da1f49-e013-4e3e-a139-99d4ae6d668f-kube-api-access-djnnw\") pod \"payload-pre-processing-e2e-worker-w0-b72d79-77d45dc4f5-nzd72\" (UID: \"50da1f49-e013-4e3e-a139-99d4ae6d668f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b72d79-77d45dc4f5-nzd72" Oct 30 07:08:35.520332 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.520211 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50da1f49-e013-4e3e-a139-99d4ae6d668f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b72d79-77d45dc4f5-nzd72\" (UID: \"50da1f49-e013-4e3e-a139-99d4ae6d668f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b72d79-77d45dc4f5-nzd72" Oct 30 07:08:35.521008 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.520972 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50da1f49-e013-4e3e-a139-99d4ae6d668f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b72d79-77d45dc4f5-nzd72\" (UID: \"50da1f49-e013-4e3e-a139-99d4ae6d668f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b72d79-77d45dc4f5-nzd72" Oct 30 07:08:35.522950 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.522926 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b72d79-7fd54fd9d4-hfflc" Oct 30 07:08:35.543126 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.543102 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djnnw\" (UniqueName: \"kubernetes.io/projected/50da1f49-e013-4e3e-a139-99d4ae6d668f-kube-api-access-djnnw\") pod \"payload-pre-processing-e2e-worker-w0-b72d79-77d45dc4f5-nzd72\" (UID: \"50da1f49-e013-4e3e-a139-99d4ae6d668f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b72d79-77d45dc4f5-nzd72" Oct 30 07:08:35.579599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.579277 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b72d79-77d45dc4f5-nzd72" Oct 30 07:08:35.589666 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.589636 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf"] Oct 30 07:08:35.596293 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.596133 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf" Oct 30 07:08:35.614496 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.613244 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf"] Oct 30 07:08:35.614496 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.613933 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-2f845b-serving-cert\"" Oct 30 07:08:35.721633 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.721601 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc75560d-af65-4bad-9bd4-c7e7156882be-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf\" (UID: \"dc75560d-af65-4bad-9bd4-c7e7156882be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf" Oct 30 07:08:35.721768 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.721717 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc75560d-af65-4bad-9bd4-c7e7156882be-metrics-certs\") pod \"maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf\" (UID: \"dc75560d-af65-4bad-9bd4-c7e7156882be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf" Oct 30 07:08:35.721768 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.721760 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhwpg\" (UniqueName: \"kubernetes.io/projected/dc75560d-af65-4bad-9bd4-c7e7156882be-kube-api-access-hhwpg\") pod \"maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf\" (UID: \"dc75560d-af65-4bad-9bd4-c7e7156882be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf" Oct 30 07:08:35.721844 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.721794 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc75560d-af65-4bad-9bd4-c7e7156882be-maas-api-tls\") pod \"maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf\" (UID: \"dc75560d-af65-4bad-9bd4-c7e7156882be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf" Oct 30 07:08:35.822461 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.822382 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc75560d-af65-4bad-9bd4-c7e7156882be-maas-api-tls\") pod \"maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf\" (UID: \"dc75560d-af65-4bad-9bd4-c7e7156882be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf" Oct 30 07:08:35.822705 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.822470 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc75560d-af65-4bad-9bd4-c7e7156882be-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf\" (UID: \"dc75560d-af65-4bad-9bd4-c7e7156882be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf" Oct 30 07:08:35.822705 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.822524 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc75560d-af65-4bad-9bd4-c7e7156882be-metrics-certs\") pod \"maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf\" (UID: \"dc75560d-af65-4bad-9bd4-c7e7156882be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf" Oct 30 07:08:35.822705 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.822562 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhwpg\" (UniqueName: \"kubernetes.io/projected/dc75560d-af65-4bad-9bd4-c7e7156882be-kube-api-access-hhwpg\") pod \"maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf\" (UID: \"dc75560d-af65-4bad-9bd4-c7e7156882be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf" Oct 30 07:08:35.822705 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.822661 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b72d79-7fd54fd9d4-hfflc"] Oct 30 07:08:35.823447 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.823411 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc75560d-af65-4bad-9bd4-c7e7156882be-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf\" (UID: \"dc75560d-af65-4bad-9bd4-c7e7156882be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf" Oct 30 07:08:35.825024 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.825001 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc75560d-af65-4bad-9bd4-c7e7156882be-maas-api-tls\") pod \"maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf\" (UID: \"dc75560d-af65-4bad-9bd4-c7e7156882be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf" Oct 30 07:08:35.825166 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.825151 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc75560d-af65-4bad-9bd4-c7e7156882be-metrics-certs\") pod \"maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf\" (UID: \"dc75560d-af65-4bad-9bd4-c7e7156882be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf" Oct 30 07:08:35.825751 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:08:35.825725 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd68fbab_46a8_4bec_8c75_e6d3e07bf691.slice/crio-a1d14e32c3fbdaa56ea32cb60f1270609e14b5172e1ced9ee578e7c65fa79a62 WatchSource:0}: Error finding container a1d14e32c3fbdaa56ea32cb60f1270609e14b5172e1ced9ee578e7c65fa79a62: Status 404 returned error can't find the container with id a1d14e32c3fbdaa56ea32cb60f1270609e14b5172e1ced9ee578e7c65fa79a62 Oct 30 07:08:35.866735 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.865718 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhwpg\" (UniqueName: \"kubernetes.io/projected/dc75560d-af65-4bad-9bd4-c7e7156882be-kube-api-access-hhwpg\") pod \"maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf\" (UID: \"dc75560d-af65-4bad-9bd4-c7e7156882be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf" Oct 30 07:08:35.887633 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.887561 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:08:35.913729 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.913703 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b72d79-77d45dc4f5-nzd72"] Oct 30 07:08:35.915651 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:08:35.915617 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50da1f49_e013_4e3e_a139_99d4ae6d668f.slice/crio-2c0464fabddd5c33b0e24636830fccff5b6acddbf2d105ce70b5e1ce07bb9291 WatchSource:0}: Error finding container 2c0464fabddd5c33b0e24636830fccff5b6acddbf2d105ce70b5e1ce07bb9291: Status 404 returned error can't find the container with id 2c0464fabddd5c33b0e24636830fccff5b6acddbf2d105ce70b5e1ce07bb9291 Oct 30 07:08:35.927833 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:35.927804 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf" Oct 30 07:08:36.230537 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:36.230508 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf"] Oct 30 07:08:36.232462 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:08:36.232431 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc75560d_af65_4bad_9bd4_c7e7156882be.slice/crio-32d1365b704d6f0b4bc6065b171d3e90c23a7372d29819a988d06797e5ec210d WatchSource:0}: Error finding container 32d1365b704d6f0b4bc6065b171d3e90c23a7372d29819a988d06797e5ec210d: Status 404 returned error can't find the container with id 32d1365b704d6f0b4bc6065b171d3e90c23a7372d29819a988d06797e5ec210d Oct 30 07:08:36.313550 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:36.313500 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e5fa2d-6dd56c7bf6-96xb9"] Oct 30 07:08:36.338329 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:36.338298 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e5fa2d-6dd56c7bf6-96xb9"] Oct 30 07:08:36.338667 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:36.338644 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e5fa2d-6dd56c7bf6-96xb9" Oct 30 07:08:36.360616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:36.358061 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-e5fa2d\"" Oct 30 07:08:36.375438 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:36.375403 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf" event={"ID":"dc75560d-af65-4bad-9bd4-c7e7156882be","Type":"ContainerStarted","Data":"32d1365b704d6f0b4bc6065b171d3e90c23a7372d29819a988d06797e5ec210d"} Oct 30 07:08:36.381244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:36.380961 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b72d79-77d45dc4f5-nzd72" event={"ID":"50da1f49-e013-4e3e-a139-99d4ae6d668f","Type":"ContainerStarted","Data":"5183599832cb3ea3108cba2db7030105f973ac4f09a02934fbc8711daa3c3b17"} Oct 30 07:08:36.381460 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:36.381442 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b72d79-77d45dc4f5-nzd72" Oct 30 07:08:36.381686 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:36.381668 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b72d79-77d45dc4f5-nzd72" event={"ID":"50da1f49-e013-4e3e-a139-99d4ae6d668f","Type":"ContainerStarted","Data":"2c0464fabddd5c33b0e24636830fccff5b6acddbf2d105ce70b5e1ce07bb9291"} Oct 30 07:08:36.383994 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:36.383968 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b72d79-7fd54fd9d4-hfflc" event={"ID":"cd68fbab-46a8-4bec-8c75-e6d3e07bf691","Type":"ContainerStarted","Data":"079f1d88b4a22bca2ddc4439ade09f3c4b43da61dfe6ae7c361394c2e20dbff3"} Oct 30 07:08:36.384097 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:36.384011 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b72d79-7fd54fd9d4-hfflc" event={"ID":"cd68fbab-46a8-4bec-8c75-e6d3e07bf691","Type":"ContainerStarted","Data":"a1d14e32c3fbdaa56ea32cb60f1270609e14b5172e1ced9ee578e7c65fa79a62"} Oct 30 07:08:36.384097 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:36.384046 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:08:36.384097 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:36.384074 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-b72d79-7fd54fd9d4-hfflc" Oct 30 07:08:36.385315 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:36.385291 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:08:36.420457 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:36.420406 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b72d79-77d45dc4f5-nzd72" podStartSLOduration=1.420395069 podStartE2EDuration="1.420395069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:08:36.418520803 +0000 UTC m=+722.761568783" watchObservedRunningTime="2026-10-30 07:08:36.420395069 +0000 UTC m=+722.763443033" Oct 30 07:08:36.437646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:36.437620 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d756\" (UniqueName: \"kubernetes.io/projected/51ace3b8-b759-4dd3-bc27-be16dcdc9d97-kube-api-access-7d756\") pod \"payload-processing-e2e-worker-w2-e5fa2d-6dd56c7bf6-96xb9\" (UID: \"51ace3b8-b759-4dd3-bc27-be16dcdc9d97\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e5fa2d-6dd56c7bf6-96xb9" Oct 30 07:08:36.437758 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:36.437667 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51ace3b8-b759-4dd3-bc27-be16dcdc9d97-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e5fa2d-6dd56c7bf6-96xb9\" (UID: \"51ace3b8-b759-4dd3-bc27-be16dcdc9d97\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e5fa2d-6dd56c7bf6-96xb9" Oct 30 07:08:36.443304 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:36.443283 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-e5fa2d-dockercfg-5hppf\"" Oct 30 07:08:36.500889 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:36.500823 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-b72d79-7fd54fd9d4-hfflc" podStartSLOduration=1.500803165 podStartE2EDuration="1.500803165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:08:36.499831665 +0000 UTC m=+722.842879632" watchObservedRunningTime="2026-10-30 07:08:36.500803165 +0000 UTC m=+722.843851131" Oct 30 07:08:36.538665 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:36.538624 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d756\" (UniqueName: \"kubernetes.io/projected/51ace3b8-b759-4dd3-bc27-be16dcdc9d97-kube-api-access-7d756\") pod \"payload-processing-e2e-worker-w2-e5fa2d-6dd56c7bf6-96xb9\" (UID: \"51ace3b8-b759-4dd3-bc27-be16dcdc9d97\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e5fa2d-6dd56c7bf6-96xb9" Oct 30 07:08:36.538846 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:36.538697 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51ace3b8-b759-4dd3-bc27-be16dcdc9d97-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e5fa2d-6dd56c7bf6-96xb9\" (UID: \"51ace3b8-b759-4dd3-bc27-be16dcdc9d97\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e5fa2d-6dd56c7bf6-96xb9" Oct 30 07:08:36.539752 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:36.539723 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51ace3b8-b759-4dd3-bc27-be16dcdc9d97-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e5fa2d-6dd56c7bf6-96xb9\" (UID: \"51ace3b8-b759-4dd3-bc27-be16dcdc9d97\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e5fa2d-6dd56c7bf6-96xb9" Oct 30 07:08:36.561358 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:36.561323 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d756\" (UniqueName: \"kubernetes.io/projected/51ace3b8-b759-4dd3-bc27-be16dcdc9d97-kube-api-access-7d756\") pod \"payload-processing-e2e-worker-w2-e5fa2d-6dd56c7bf6-96xb9\" (UID: \"51ace3b8-b759-4dd3-bc27-be16dcdc9d97\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e5fa2d-6dd56c7bf6-96xb9" Oct 30 07:08:36.672555 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:36.672534 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e5fa2d-6dd56c7bf6-96xb9" Oct 30 07:08:36.972269 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:36.972241 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e5fa2d-6dd56c7bf6-96xb9"] Oct 30 07:08:36.973657 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:08:36.973628 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51ace3b8_b759_4dd3_bc27_be16dcdc9d97.slice/crio-79be54ec934a7a5f9a907ad4a362344132e8fb7355ccc376e59e824b69402703 WatchSource:0}: Error finding container 79be54ec934a7a5f9a907ad4a362344132e8fb7355ccc376e59e824b69402703: Status 404 returned error can't find the container with id 79be54ec934a7a5f9a907ad4a362344132e8fb7355ccc376e59e824b69402703 Oct 30 07:08:37.390816 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:37.390731 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e5fa2d-6dd56c7bf6-96xb9" event={"ID":"51ace3b8-b759-4dd3-bc27-be16dcdc9d97","Type":"ContainerStarted","Data":"9a69722b7c209eb77333c093aaf7cde85f117f7329fafcd264d979fed50a4d71"} Oct 30 07:08:37.390963 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:37.390910 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-e5fa2d-6dd56c7bf6-96xb9" Oct 30 07:08:37.391019 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:37.390960 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e5fa2d-6dd56c7bf6-96xb9" event={"ID":"51ace3b8-b759-4dd3-bc27-be16dcdc9d97","Type":"ContainerStarted","Data":"79be54ec934a7a5f9a907ad4a362344132e8fb7355ccc376e59e824b69402703"} Oct 30 07:08:37.392458 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:37.392428 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf" event={"ID":"dc75560d-af65-4bad-9bd4-c7e7156882be","Type":"ContainerStarted","Data":"50257b1188bff7f3033d259b209a8f5057134d5378fb5582cf9e1fc287c1c053"} Oct 30 07:08:37.392663 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:37.392627 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf" Oct 30 07:08:37.406461 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:37.406438 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4"] Oct 30 07:08:37.409613 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:37.409598 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4" Oct 30 07:08:37.429263 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:37.429227 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-b72d79-serving-cert\"" Oct 30 07:08:37.446140 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:37.446079 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-e5fa2d-6dd56c7bf6-96xb9" podStartSLOduration=1.446060271 podStartE2EDuration="1.446060271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:08:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:08:37.442525773 +0000 UTC m=+723.785573739" watchObservedRunningTime="2026-10-30 07:08:37.446060271 +0000 UTC m=+723.789108238" Oct 30 07:08:37.450343 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:37.450304 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4"] Oct 30 07:08:37.533354 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:37.533296 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf" podStartSLOduration=2.112521558 podStartE2EDuration="2.533280676s" totalImagesPullingTime="420.759118ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:08:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:08:37.531600509 +0000 UTC m=+723.874648498" watchObservedRunningTime="2026-10-30 07:08:37.533280676 +0000 UTC m=+723.876328640" Oct 30 07:08:37.556222 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:37.556189 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb29167c-8d9b-4938-afb5-0c974e7e34e7-metrics-certs\") pod \"maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4\" (UID: \"bb29167c-8d9b-4938-afb5-0c974e7e34e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4" Oct 30 07:08:37.556374 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:37.556258 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb29167c-8d9b-4938-afb5-0c974e7e34e7-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4\" (UID: \"bb29167c-8d9b-4938-afb5-0c974e7e34e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4" Oct 30 07:08:37.556730 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:37.556699 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9mct\" (UniqueName: \"kubernetes.io/projected/bb29167c-8d9b-4938-afb5-0c974e7e34e7-kube-api-access-w9mct\") pod \"maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4\" (UID: \"bb29167c-8d9b-4938-afb5-0c974e7e34e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4" Oct 30 07:08:37.557019 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:37.556999 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb29167c-8d9b-4938-afb5-0c974e7e34e7-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4\" (UID: \"bb29167c-8d9b-4938-afb5-0c974e7e34e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4" Oct 30 07:08:37.659671 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:37.659146 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb29167c-8d9b-4938-afb5-0c974e7e34e7-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4\" (UID: \"bb29167c-8d9b-4938-afb5-0c974e7e34e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4" Oct 30 07:08:37.659671 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:37.659244 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb29167c-8d9b-4938-afb5-0c974e7e34e7-metrics-certs\") pod \"maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4\" (UID: \"bb29167c-8d9b-4938-afb5-0c974e7e34e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4" Oct 30 07:08:37.659671 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:37.659286 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb29167c-8d9b-4938-afb5-0c974e7e34e7-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4\" (UID: \"bb29167c-8d9b-4938-afb5-0c974e7e34e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4" Oct 30 07:08:37.659671 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:37.659347 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9mct\" (UniqueName: \"kubernetes.io/projected/bb29167c-8d9b-4938-afb5-0c974e7e34e7-kube-api-access-w9mct\") pod \"maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4\" (UID: \"bb29167c-8d9b-4938-afb5-0c974e7e34e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4" Oct 30 07:08:37.660411 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:37.660340 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb29167c-8d9b-4938-afb5-0c974e7e34e7-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4\" (UID: \"bb29167c-8d9b-4938-afb5-0c974e7e34e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4" Oct 30 07:08:37.676328 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:37.671516 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb29167c-8d9b-4938-afb5-0c974e7e34e7-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4\" (UID: \"bb29167c-8d9b-4938-afb5-0c974e7e34e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4" Oct 30 07:08:37.676899 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:37.676800 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb29167c-8d9b-4938-afb5-0c974e7e34e7-metrics-certs\") pod \"maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4\" (UID: \"bb29167c-8d9b-4938-afb5-0c974e7e34e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4" Oct 30 07:08:37.693615 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:37.693567 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9mct\" (UniqueName: \"kubernetes.io/projected/bb29167c-8d9b-4938-afb5-0c974e7e34e7-kube-api-access-w9mct\") pod \"maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4\" (UID: \"bb29167c-8d9b-4938-afb5-0c974e7e34e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4" Oct 30 07:08:37.722509 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:37.722475 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4" Oct 30 07:08:38.025931 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:38.025901 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4"] Oct 30 07:08:38.027067 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:08:38.027036 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb29167c_8d9b_4938_afb5_0c974e7e34e7.slice/crio-bf7b95cdf69ca8af9e3cd59d02cf6f3c3e77188b06b4bab8022acad936e07950 WatchSource:0}: Error finding container bf7b95cdf69ca8af9e3cd59d02cf6f3c3e77188b06b4bab8022acad936e07950: Status 404 returned error can't find the container with id bf7b95cdf69ca8af9e3cd59d02cf6f3c3e77188b06b4bab8022acad936e07950 Oct 30 07:08:38.398409 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:38.398309 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4" event={"ID":"bb29167c-8d9b-4938-afb5-0c974e7e34e7","Type":"ContainerStarted","Data":"bf7b95cdf69ca8af9e3cd59d02cf6f3c3e77188b06b4bab8022acad936e07950"} Oct 30 07:08:39.405249 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:39.405207 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4" event={"ID":"bb29167c-8d9b-4938-afb5-0c974e7e34e7","Type":"ContainerStarted","Data":"da0add70e687fc05490cbe8469cba5140027d4edde3f71c0b6448b23f1ca5fe5"} Oct 30 07:08:39.405773 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:39.405362 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4" Oct 30 07:08:39.450364 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:39.450307 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4" podStartSLOduration=2.046741408 podStartE2EDuration="2.4502918s" totalImagesPullingTime="403.550392ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:08:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:08:39.448348719 +0000 UTC m=+725.791396681" watchObservedRunningTime="2026-10-30 07:08:39.4502918 +0000 UTC m=+725.793339812" Oct 30 07:08:43.404138 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:43.404105 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf" Oct 30 07:08:45.366733 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:45.366702 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-2f845b-7d977fd55d-5ctj5" Oct 30 07:08:45.367176 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:45.366779 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2f845b-5d5d94cf95-jsspt" Oct 30 07:08:45.416903 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:45.416872 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4" Oct 30 07:08:48.402237 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:48.402180 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b72d79-77d45dc4f5-nzd72" Oct 30 07:08:48.402691 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:48.402353 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-e5fa2d-6dd56c7bf6-96xb9" Oct 30 07:08:48.402691 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:48.402411 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-b72d79-7fd54fd9d4-hfflc" Oct 30 07:08:52.504069 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:52.504035 4951 generic.go:356] "Generic (PLEG): container finished" podID="42ea0497-3b5f-4c33-a53a-bd636a4379ba" containerID="17f52bcda66d0ed800ec1c3ef785437e87ec0ec6dae43053a48e1cbce9f9de28" exitCode=0 Oct 30 07:08:52.504511 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:52.504107 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" event={"ID":"42ea0497-3b5f-4c33-a53a-bd636a4379ba","Type":"ContainerDied","Data":"17f52bcda66d0ed800ec1c3ef785437e87ec0ec6dae43053a48e1cbce9f9de28"} Oct 30 07:08:52.504600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:52.504523 4951 scope.go:122] "RemoveContainer" containerID="17f52bcda66d0ed800ec1c3ef785437e87ec0ec6dae43053a48e1cbce9f9de28" Oct 30 07:08:52.505973 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:52.505930 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:08:52.506141 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:52.506010 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:08:52.506141 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:52.506046 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:08:53.057895 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:08:53.057853 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:08:53.516034 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:53.515995 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" event={"ID":"42ea0497-3b5f-4c33-a53a-bd636a4379ba","Type":"ContainerStarted","Data":"0c79da0deb4539c1b8607b87e63d6be137320dc2ffe3ebe69a76feaed8c732f0"} Oct 30 07:08:53.571605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:53.571430 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:53.571811 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:53.571628 4951 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 07:08:53.571811 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:53.571693 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" podUID="42ea0497-3b5f-4c33-a53a-bd636a4379ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 07:08:54.566601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:54.565988 4951 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 07:08:54.566601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:54.566047 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" podUID="42ea0497-3b5f-4c33-a53a-bd636a4379ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 07:08:55.566596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:55.566037 4951 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 07:08:55.566596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:55.566103 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" podUID="42ea0497-3b5f-4c33-a53a-bd636a4379ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 07:08:56.563419 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:56.563380 4951 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 07:08:56.563601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:56.563445 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" podUID="42ea0497-3b5f-4c33-a53a-bd636a4379ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 07:08:57.563298 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:57.563262 4951 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 07:08:57.563820 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:57.563325 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" podUID="42ea0497-3b5f-4c33-a53a-bd636a4379ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 07:08:58.563993 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:58.563956 4951 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 07:08:58.564487 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:58.564011 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" podUID="42ea0497-3b5f-4c33-a53a-bd636a4379ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 07:08:59.567917 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:59.567842 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:59.568487 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:59.568456 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:08:59.569562 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:08:59.569532 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:09:00.580914 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:00.580872 4951 generic.go:356] "Generic (PLEG): container finished" podID="de840f64-9d0c-4e5b-aa38-19c054fc8b17" containerID="b71ecc855f084d3301e8e09f292911517600b11035063e209cff52d778f237c0" exitCode=0 Oct 30 07:09:00.581400 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:00.580946 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" event={"ID":"de840f64-9d0c-4e5b-aa38-19c054fc8b17","Type":"ContainerDied","Data":"b71ecc855f084d3301e8e09f292911517600b11035063e209cff52d778f237c0"} Oct 30 07:09:00.581566 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:00.581550 4951 scope.go:122] "RemoveContainer" containerID="b71ecc855f084d3301e8e09f292911517600b11035063e209cff52d778f237c0" Oct 30 07:09:00.582809 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:00.582776 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:09:00.582926 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:00.582860 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:09:00.582926 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:00.582909 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:09:01.588205 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:01.588164 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" event={"ID":"de840f64-9d0c-4e5b-aa38-19c054fc8b17","Type":"ContainerStarted","Data":"9c23322f120088c96aa4b1eb87191910c7dde6908aae780249ca9cbb3d9b9ac6"} Oct 30 07:09:01.881918 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:01.881822 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:09:01.883329 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:01.883300 4951 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 07:09:01.883475 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:01.883347 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" podUID="de840f64-9d0c-4e5b-aa38-19c054fc8b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 07:09:02.882199 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:02.882163 4951 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 07:09:02.882647 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:02.882229 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" podUID="de840f64-9d0c-4e5b-aa38-19c054fc8b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 07:09:03.620827 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:03.620794 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct"] Oct 30 07:09:03.653258 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:03.653227 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct"] Oct 30 07:09:03.653556 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:03.653539 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:09:03.671613 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:03.671587 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-tls-certs\") pod \"simulator-w0-b72d79-kserve-697678f9b-vmbct\" (UID: \"fab18d8f-7bf2-44eb-a099-fd2a60990d0a\") " pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:09:03.671805 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:03.671780 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwxx7\" (UniqueName: \"kubernetes.io/projected/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-kube-api-access-qwxx7\") pod \"simulator-w0-b72d79-kserve-697678f9b-vmbct\" (UID: \"fab18d8f-7bf2-44eb-a099-fd2a60990d0a\") " pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:09:03.671931 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:03.671903 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-model-cache\") pod \"simulator-w0-b72d79-kserve-697678f9b-vmbct\" (UID: \"fab18d8f-7bf2-44eb-a099-fd2a60990d0a\") " pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:09:03.672028 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:03.671945 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-tmp-dir\") pod \"simulator-w0-b72d79-kserve-697678f9b-vmbct\" (UID: \"fab18d8f-7bf2-44eb-a099-fd2a60990d0a\") " pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:09:03.672072 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:03.672040 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-home\") pod \"simulator-w0-b72d79-kserve-697678f9b-vmbct\" (UID: \"fab18d8f-7bf2-44eb-a099-fd2a60990d0a\") " pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:09:03.672113 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:03.672093 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-dshm\") pod \"simulator-w0-b72d79-kserve-697678f9b-vmbct\" (UID: \"fab18d8f-7bf2-44eb-a099-fd2a60990d0a\") " pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:09:03.679117 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:03.679012 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b72d79\"/\"openshift-service-ca.crt\"" Oct 30 07:09:03.680548 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:03.680525 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b72d79\"/\"simulator-w0-b72d79-kserve-self-signed-certs\"" Oct 30 07:09:03.680711 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:03.680555 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b72d79\"/\"kube-root-ca.crt\"" Oct 30 07:09:03.680711 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:03.680565 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b72d79\"/\"default-dockercfg-zzm9r\"" Oct 30 07:09:03.772766 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:03.772721 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-home\") pod \"simulator-w0-b72d79-kserve-697678f9b-vmbct\" (UID: \"fab18d8f-7bf2-44eb-a099-fd2a60990d0a\") " pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:09:03.772955 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:03.772792 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-dshm\") pod \"simulator-w0-b72d79-kserve-697678f9b-vmbct\" (UID: \"fab18d8f-7bf2-44eb-a099-fd2a60990d0a\") " pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:09:03.772955 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:03.772866 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-tls-certs\") pod \"simulator-w0-b72d79-kserve-697678f9b-vmbct\" (UID: \"fab18d8f-7bf2-44eb-a099-fd2a60990d0a\") " pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:09:03.772955 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:03.772905 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwxx7\" (UniqueName: \"kubernetes.io/projected/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-kube-api-access-qwxx7\") pod \"simulator-w0-b72d79-kserve-697678f9b-vmbct\" (UID: \"fab18d8f-7bf2-44eb-a099-fd2a60990d0a\") " pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:09:03.773150 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:03.772988 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-model-cache\") pod \"simulator-w0-b72d79-kserve-697678f9b-vmbct\" (UID: \"fab18d8f-7bf2-44eb-a099-fd2a60990d0a\") " pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:09:03.773150 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:03.773009 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-tmp-dir\") pod \"simulator-w0-b72d79-kserve-697678f9b-vmbct\" (UID: \"fab18d8f-7bf2-44eb-a099-fd2a60990d0a\") " pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:09:03.773251 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:03.773177 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-home\") pod \"simulator-w0-b72d79-kserve-697678f9b-vmbct\" (UID: \"fab18d8f-7bf2-44eb-a099-fd2a60990d0a\") " pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:09:03.773753 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:03.773716 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-model-cache\") pod \"simulator-w0-b72d79-kserve-697678f9b-vmbct\" (UID: \"fab18d8f-7bf2-44eb-a099-fd2a60990d0a\") " pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:09:03.773875 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:03.773722 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-tmp-dir\") pod \"simulator-w0-b72d79-kserve-697678f9b-vmbct\" (UID: \"fab18d8f-7bf2-44eb-a099-fd2a60990d0a\") " pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:09:03.775765 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:03.775191 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-dshm\") pod \"simulator-w0-b72d79-kserve-697678f9b-vmbct\" (UID: \"fab18d8f-7bf2-44eb-a099-fd2a60990d0a\") " pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:09:03.775765 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:03.775712 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-tls-certs\") pod \"simulator-w0-b72d79-kserve-697678f9b-vmbct\" (UID: \"fab18d8f-7bf2-44eb-a099-fd2a60990d0a\") " pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:09:03.795404 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:03.795379 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwxx7\" (UniqueName: \"kubernetes.io/projected/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-kube-api-access-qwxx7\") pod \"simulator-w0-b72d79-kserve-697678f9b-vmbct\" (UID: \"fab18d8f-7bf2-44eb-a099-fd2a60990d0a\") " pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:09:03.886493 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:03.886007 4951 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 07:09:03.886493 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:03.886070 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" podUID="de840f64-9d0c-4e5b-aa38-19c054fc8b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 07:09:03.974839 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:03.974791 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:09:04.291769 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:04.291740 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct"] Oct 30 07:09:04.294438 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:09:04.294401 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfab18d8f_7bf2_44eb_a099_fd2a60990d0a.slice/crio-5dcd17ffeec2036f8fd7616817f705b12c04e1ff93ad6e0ae0dce52280ee6eaa WatchSource:0}: Error finding container 5dcd17ffeec2036f8fd7616817f705b12c04e1ff93ad6e0ae0dce52280ee6eaa: Status 404 returned error can't find the container with id 5dcd17ffeec2036f8fd7616817f705b12c04e1ff93ad6e0ae0dce52280ee6eaa Oct 30 07:09:04.605525 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:04.605430 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" event={"ID":"fab18d8f-7bf2-44eb-a099-fd2a60990d0a","Type":"ContainerStarted","Data":"7f032fb28d453ab0b355d92f034f3bb800e34c6034836b17077a71166a4d56ef"} Oct 30 07:09:04.605525 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:04.605485 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" event={"ID":"fab18d8f-7bf2-44eb-a099-fd2a60990d0a","Type":"ContainerStarted","Data":"5dcd17ffeec2036f8fd7616817f705b12c04e1ff93ad6e0ae0dce52280ee6eaa"} Oct 30 07:09:04.648845 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:04.648786 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podStartSLOduration=1.648771985 podStartE2EDuration="1.648771985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:09:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:09:04.647126485 +0000 UTC m=+750.990174481" watchObservedRunningTime="2026-10-30 07:09:04.648771985 +0000 UTC m=+750.991819949" Oct 30 07:09:04.882263 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:04.882176 4951 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 07:09:04.882263 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:04.882237 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" podUID="de840f64-9d0c-4e5b-aa38-19c054fc8b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 07:09:04.975668 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:04.975632 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:09:05.882172 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:05.882132 4951 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 07:09:05.882352 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:05.882218 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" podUID="de840f64-9d0c-4e5b-aa38-19c054fc8b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 07:09:06.384844 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:06.384813 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:09:06.885461 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:06.885426 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:09:06.886383 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:06.886360 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:09:09.321367 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:09.321332 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr"] Oct 30 07:09:09.326290 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:09.326266 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:09:09.353399 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:09.353366 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a0b3f5\"/\"openshift-service-ca.crt\"" Oct 30 07:09:09.353558 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:09.353481 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a0b3f5\"/\"kube-root-ca.crt\"" Oct 30 07:09:09.353558 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:09.353553 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a0b3f5\"/\"default-dockercfg-thbxc\"" Oct 30 07:09:09.353685 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:09.353634 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a0b3f5\"/\"simulator-w1-a0b3f5-kserve-self-signed-certs\"" Oct 30 07:09:09.355781 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:09.355756 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr"] Oct 30 07:09:09.444249 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:09.444211 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm6sf\" (UniqueName: \"kubernetes.io/projected/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-kube-api-access-vm6sf\") pod \"simulator-w1-a0b3f5-kserve-767968f6cc-nklmr\" (UID: \"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106\") " pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:09:09.444449 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:09.444272 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-tls-certs\") pod \"simulator-w1-a0b3f5-kserve-767968f6cc-nklmr\" (UID: \"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106\") " pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:09:09.444449 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:09.444345 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-home\") pod \"simulator-w1-a0b3f5-kserve-767968f6cc-nklmr\" (UID: \"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106\") " pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:09:09.444449 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:09.444397 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-tmp-dir\") pod \"simulator-w1-a0b3f5-kserve-767968f6cc-nklmr\" (UID: \"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106\") " pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:09:09.444608 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:09.444465 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-dshm\") pod \"simulator-w1-a0b3f5-kserve-767968f6cc-nklmr\" (UID: \"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106\") " pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:09:09.444608 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:09.444535 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-model-cache\") pod \"simulator-w1-a0b3f5-kserve-767968f6cc-nklmr\" (UID: \"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106\") " pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:09:09.548904 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:09.548859 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-model-cache\") pod \"simulator-w1-a0b3f5-kserve-767968f6cc-nklmr\" (UID: \"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106\") " pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:09:09.549137 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:09.548966 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm6sf\" (UniqueName: \"kubernetes.io/projected/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-kube-api-access-vm6sf\") pod \"simulator-w1-a0b3f5-kserve-767968f6cc-nklmr\" (UID: \"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106\") " pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:09:09.549137 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:09.548997 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-tls-certs\") pod \"simulator-w1-a0b3f5-kserve-767968f6cc-nklmr\" (UID: \"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106\") " pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:09:09.549137 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:09.549065 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-home\") pod \"simulator-w1-a0b3f5-kserve-767968f6cc-nklmr\" (UID: \"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106\") " pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:09:09.549137 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:09.549120 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-tmp-dir\") pod \"simulator-w1-a0b3f5-kserve-767968f6cc-nklmr\" (UID: \"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106\") " pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:09:09.549369 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:09.549157 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-dshm\") pod \"simulator-w1-a0b3f5-kserve-767968f6cc-nklmr\" (UID: \"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106\") " pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:09:09.565269 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:09.550676 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-home\") pod \"simulator-w1-a0b3f5-kserve-767968f6cc-nklmr\" (UID: \"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106\") " pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:09:09.565269 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:09.550947 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-tmp-dir\") pod \"simulator-w1-a0b3f5-kserve-767968f6cc-nklmr\" (UID: \"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106\") " pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:09:09.565893 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:09.565850 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-model-cache\") pod \"simulator-w1-a0b3f5-kserve-767968f6cc-nklmr\" (UID: \"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106\") " pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:09:09.571126 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:09.571092 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-dshm\") pod \"simulator-w1-a0b3f5-kserve-767968f6cc-nklmr\" (UID: \"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106\") " pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:09:09.573601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:09.573504 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-tls-certs\") pod \"simulator-w1-a0b3f5-kserve-767968f6cc-nklmr\" (UID: \"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106\") " pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:09:09.588036 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:09.587992 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm6sf\" (UniqueName: \"kubernetes.io/projected/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-kube-api-access-vm6sf\") pod \"simulator-w1-a0b3f5-kserve-767968f6cc-nklmr\" (UID: \"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106\") " pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:09:09.641157 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:09.641109 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:09:09.942764 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:09.942730 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr"] Oct 30 07:09:09.944282 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:09:09.944243 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fcbe6d6_ae98_48cf_b62f_c14e0f7f3106.slice/crio-035c8e774135ed0908771cbc23f0ee6bc31431cf50812aa5acbb0693a0bd6d53 WatchSource:0}: Error finding container 035c8e774135ed0908771cbc23f0ee6bc31431cf50812aa5acbb0693a0bd6d53: Status 404 returned error can't find the container with id 035c8e774135ed0908771cbc23f0ee6bc31431cf50812aa5acbb0693a0bd6d53 Oct 30 07:09:10.638272 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:10.638232 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" event={"ID":"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106","Type":"ContainerStarted","Data":"7d073a0f59654b3d4dcef450f56fbb90ee73060b644a13a569407a878f68635e"} Oct 30 07:09:10.638272 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:10.638281 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" event={"ID":"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106","Type":"ContainerStarted","Data":"035c8e774135ed0908771cbc23f0ee6bc31431cf50812aa5acbb0693a0bd6d53"} Oct 30 07:09:10.641551 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:10.641530 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:09:10.679397 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:10.679346 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" podStartSLOduration=1.6793315359999998 podStartE2EDuration="1.679331536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:09:10.677445108 +0000 UTC m=+757.020493074" watchObservedRunningTime="2026-10-30 07:09:10.679331536 +0000 UTC m=+757.022379516" Oct 30 07:09:13.975935 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:13.975892 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:09:13.994927 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:13.994901 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:09:14.682459 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:14.682428 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:09:19.642232 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:19.642129 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:09:19.660277 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:19.660243 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:09:19.715239 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:19.715203 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:09:23.025308 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:09:23.025266 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:09:35.346228 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:35.346187 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk"] Oct 30 07:09:35.350837 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:35.350804 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:09:35.368699 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:35.368667 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk"] Oct 30 07:09:35.377051 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:35.377023 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b72d79\"/\"premium-w0-b72d79-kserve-self-signed-certs\"" Oct 30 07:09:35.471139 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:35.471097 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psh6f\" (UniqueName: \"kubernetes.io/projected/d680a08d-af67-455d-87bc-9fa6a26ad37a-kube-api-access-psh6f\") pod \"premium-w0-b72d79-kserve-7d9886d56f-dfwsk\" (UID: \"d680a08d-af67-455d-87bc-9fa6a26ad37a\") " pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:09:35.471320 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:35.471230 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d680a08d-af67-455d-87bc-9fa6a26ad37a-model-cache\") pod \"premium-w0-b72d79-kserve-7d9886d56f-dfwsk\" (UID: \"d680a08d-af67-455d-87bc-9fa6a26ad37a\") " pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:09:35.471320 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:35.471279 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d680a08d-af67-455d-87bc-9fa6a26ad37a-tls-certs\") pod \"premium-w0-b72d79-kserve-7d9886d56f-dfwsk\" (UID: \"d680a08d-af67-455d-87bc-9fa6a26ad37a\") " pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:09:35.471320 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:35.471305 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d680a08d-af67-455d-87bc-9fa6a26ad37a-home\") pod \"premium-w0-b72d79-kserve-7d9886d56f-dfwsk\" (UID: \"d680a08d-af67-455d-87bc-9fa6a26ad37a\") " pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:09:35.471470 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:35.471455 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d680a08d-af67-455d-87bc-9fa6a26ad37a-tmp-dir\") pod \"premium-w0-b72d79-kserve-7d9886d56f-dfwsk\" (UID: \"d680a08d-af67-455d-87bc-9fa6a26ad37a\") " pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:09:35.471507 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:35.471495 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d680a08d-af67-455d-87bc-9fa6a26ad37a-dshm\") pod \"premium-w0-b72d79-kserve-7d9886d56f-dfwsk\" (UID: \"d680a08d-af67-455d-87bc-9fa6a26ad37a\") " pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:09:35.572851 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:35.572811 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d680a08d-af67-455d-87bc-9fa6a26ad37a-tmp-dir\") pod \"premium-w0-b72d79-kserve-7d9886d56f-dfwsk\" (UID: \"d680a08d-af67-455d-87bc-9fa6a26ad37a\") " pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:09:35.573608 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:35.573561 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d680a08d-af67-455d-87bc-9fa6a26ad37a-dshm\") pod \"premium-w0-b72d79-kserve-7d9886d56f-dfwsk\" (UID: \"d680a08d-af67-455d-87bc-9fa6a26ad37a\") " pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:09:35.573823 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:35.573806 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psh6f\" (UniqueName: \"kubernetes.io/projected/d680a08d-af67-455d-87bc-9fa6a26ad37a-kube-api-access-psh6f\") pod \"premium-w0-b72d79-kserve-7d9886d56f-dfwsk\" (UID: \"d680a08d-af67-455d-87bc-9fa6a26ad37a\") " pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:09:35.573986 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:35.573972 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d680a08d-af67-455d-87bc-9fa6a26ad37a-model-cache\") pod \"premium-w0-b72d79-kserve-7d9886d56f-dfwsk\" (UID: \"d680a08d-af67-455d-87bc-9fa6a26ad37a\") " pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:09:35.574110 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:35.574098 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d680a08d-af67-455d-87bc-9fa6a26ad37a-tls-certs\") pod \"premium-w0-b72d79-kserve-7d9886d56f-dfwsk\" (UID: \"d680a08d-af67-455d-87bc-9fa6a26ad37a\") " pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:09:35.574227 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:35.574213 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d680a08d-af67-455d-87bc-9fa6a26ad37a-home\") pod \"premium-w0-b72d79-kserve-7d9886d56f-dfwsk\" (UID: \"d680a08d-af67-455d-87bc-9fa6a26ad37a\") " pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:09:35.574702 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:35.574682 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d680a08d-af67-455d-87bc-9fa6a26ad37a-home\") pod \"premium-w0-b72d79-kserve-7d9886d56f-dfwsk\" (UID: \"d680a08d-af67-455d-87bc-9fa6a26ad37a\") " pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:09:35.574828 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:35.573489 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d680a08d-af67-455d-87bc-9fa6a26ad37a-tmp-dir\") pod \"premium-w0-b72d79-kserve-7d9886d56f-dfwsk\" (UID: \"d680a08d-af67-455d-87bc-9fa6a26ad37a\") " pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:09:35.575563 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:35.575439 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d680a08d-af67-455d-87bc-9fa6a26ad37a-model-cache\") pod \"premium-w0-b72d79-kserve-7d9886d56f-dfwsk\" (UID: \"d680a08d-af67-455d-87bc-9fa6a26ad37a\") " pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:09:35.581636 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:35.581615 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d680a08d-af67-455d-87bc-9fa6a26ad37a-tls-certs\") pod \"premium-w0-b72d79-kserve-7d9886d56f-dfwsk\" (UID: \"d680a08d-af67-455d-87bc-9fa6a26ad37a\") " pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:09:35.582546 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:35.582528 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d680a08d-af67-455d-87bc-9fa6a26ad37a-dshm\") pod \"premium-w0-b72d79-kserve-7d9886d56f-dfwsk\" (UID: \"d680a08d-af67-455d-87bc-9fa6a26ad37a\") " pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:09:35.598031 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:35.597980 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psh6f\" (UniqueName: \"kubernetes.io/projected/d680a08d-af67-455d-87bc-9fa6a26ad37a-kube-api-access-psh6f\") pod \"premium-w0-b72d79-kserve-7d9886d56f-dfwsk\" (UID: \"d680a08d-af67-455d-87bc-9fa6a26ad37a\") " pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:09:35.665023 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:35.664991 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:09:35.967683 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:35.967650 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk"] Oct 30 07:09:35.968289 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:09:35.968261 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd680a08d_af67_455d_87bc_9fa6a26ad37a.slice/crio-cc98cb2d8be4411ca0ac05d5341722730f3b572e9f8bc03cf3bbfb69a5b37154 WatchSource:0}: Error finding container cc98cb2d8be4411ca0ac05d5341722730f3b572e9f8bc03cf3bbfb69a5b37154: Status 404 returned error can't find the container with id cc98cb2d8be4411ca0ac05d5341722730f3b572e9f8bc03cf3bbfb69a5b37154 Oct 30 07:09:36.839779 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:36.839739 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" event={"ID":"d680a08d-af67-455d-87bc-9fa6a26ad37a","Type":"ContainerStarted","Data":"6c4fa67a100d7e4764ab97b771c3999de42eac25335905cf81ca9a426c19b8fe"} Oct 30 07:09:36.840174 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:36.839792 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" event={"ID":"d680a08d-af67-455d-87bc-9fa6a26ad37a","Type":"ContainerStarted","Data":"cc98cb2d8be4411ca0ac05d5341722730f3b572e9f8bc03cf3bbfb69a5b37154"} Oct 30 07:09:36.879758 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:36.879707 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" podStartSLOduration=1.87969324 podStartE2EDuration="1.87969324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:09:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:09:36.878674447 +0000 UTC m=+783.221722434" watchObservedRunningTime="2026-10-30 07:09:36.87969324 +0000 UTC m=+783.222741205" Oct 30 07:09:37.666043 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:37.665992 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:09:45.665402 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:45.665362 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:09:45.678764 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:45.678733 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:09:45.903267 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:45.903236 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:09:53.023611 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:09:53.023554 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:09:58.838419 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:58.838375 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn"] Oct 30 07:09:58.842672 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:58.842640 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:09:58.855798 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:58.855766 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89d560f3-70af-44f7-8084-4ca608698965-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:09:58.855951 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:58.855842 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:09:58.855951 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:58.855938 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89d560f3-70af-44f7-8084-4ca608698965-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:09:58.856051 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:58.855970 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:09:58.856051 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:58.856011 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:09:58.856156 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:58.856139 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pp2v\" (UniqueName: \"kubernetes.io/projected/89d560f3-70af-44f7-8084-4ca608698965-kube-api-access-7pp2v\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:09:58.856218 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:58.856205 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:09:58.856319 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:58.856300 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:09:58.856362 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:58.856334 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89d560f3-70af-44f7-8084-4ca608698965-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:09:58.859381 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:58.859362 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e5fa2d-openshift-default-dockercfg-sppbj\"" Oct 30 07:09:58.871332 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:58.871285 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn"] Oct 30 07:09:58.957384 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:58.957348 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:09:58.957596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:58.957396 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89d560f3-70af-44f7-8084-4ca608698965-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:09:58.957596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:58.957440 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89d560f3-70af-44f7-8084-4ca608698965-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:09:58.957596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:58.957522 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:09:58.957785 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:58.957613 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89d560f3-70af-44f7-8084-4ca608698965-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:09:58.957785 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:58.957653 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:09:58.957785 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:58.957704 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:09:58.957785 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:58.957735 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pp2v\" (UniqueName: \"kubernetes.io/projected/89d560f3-70af-44f7-8084-4ca608698965-kube-api-access-7pp2v\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:09:58.957994 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:58.957789 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:09:58.957994 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:58.957815 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:09:58.958094 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:58.958048 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:09:58.958152 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:58.958129 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:09:58.958358 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:58.958333 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:09:58.958545 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:58.958525 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89d560f3-70af-44f7-8084-4ca608698965-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:09:58.961749 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:58.961722 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:09:58.962057 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:58.962036 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89d560f3-70af-44f7-8084-4ca608698965-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:09:58.980688 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:58.980662 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89d560f3-70af-44f7-8084-4ca608698965-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:09:58.980887 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:58.980869 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pp2v\" (UniqueName: \"kubernetes.io/projected/89d560f3-70af-44f7-8084-4ca608698965-kube-api-access-7pp2v\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:09:59.157819 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:59.157726 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:09:59.469957 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:59.469925 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn"] Oct 30 07:09:59.470255 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:09:59.470232 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89d560f3_70af_44f7_8084_4ca608698965.slice/crio-929ba8187bc7004209cd9b283b13dea040a50ab802a3457bf162be7b2a4242df WatchSource:0}: Error finding container 929ba8187bc7004209cd9b283b13dea040a50ab802a3457bf162be7b2a4242df: Status 404 returned error can't find the container with id 929ba8187bc7004209cd9b283b13dea040a50ab802a3457bf162be7b2a4242df Oct 30 07:09:59.472650 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:59.472620 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:09:59.472712 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:59.472681 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:09:59.472756 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:59.472710 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:09:59.972692 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:59.972655 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" event={"ID":"89d560f3-70af-44f7-8084-4ca608698965","Type":"ContainerStarted","Data":"190c321dd69ed8191538ac789bc0e5da75bdf68196e27387c762dd73f8184c76"} Oct 30 07:09:59.973113 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:09:59.972708 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" event={"ID":"89d560f3-70af-44f7-8084-4ca608698965","Type":"ContainerStarted","Data":"929ba8187bc7004209cd9b283b13dea040a50ab802a3457bf162be7b2a4242df"} Oct 30 07:10:00.158191 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:10:00.158157 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:10:00.160167 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:10:00.160138 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 07:10:00.160308 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:10:00.160198 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" podUID="89d560f3-70af-44f7-8084-4ca608698965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 07:10:01.158731 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:10:01.158679 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 07:10:01.159240 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:10:01.158749 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" podUID="89d560f3-70af-44f7-8084-4ca608698965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 07:10:02.158824 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:10:02.158783 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 07:10:02.159242 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:10:02.158840 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" podUID="89d560f3-70af-44f7-8084-4ca608698965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 07:10:03.158662 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:10:03.158621 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 07:10:03.158821 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:10:03.158703 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" podUID="89d560f3-70af-44f7-8084-4ca608698965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 07:10:04.158624 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:10:04.158587 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 07:10:04.159038 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:10:04.158644 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" podUID="89d560f3-70af-44f7-8084-4ca608698965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 07:10:05.158222 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:10:05.158184 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 07:10:05.158388 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:10:05.158248 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" podUID="89d560f3-70af-44f7-8084-4ca608698965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 07:10:06.162484 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:10:06.162453 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:10:06.162898 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:10:06.162838 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:10:06.163478 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:10:06.163456 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:10:06.217230 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:10:06.217183 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" podStartSLOduration=8.21717066 podStartE2EDuration="8.21717066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:10:00.036966215 +0000 UTC m=+806.380014234" watchObservedRunningTime="2026-10-30 07:10:06.21717066 +0000 UTC m=+812.560218622" Oct 30 07:10:23.025152 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:10:23.025100 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:10:53.023472 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:10:53.023406 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:11:13.735191 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:13.735152 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792"] Oct 30 07:11:13.739919 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:13.739890 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:11:13.754992 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:13.754958 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e5fa2d\"/\"kube-root-ca.crt\"" Oct 30 07:11:13.765981 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:13.765949 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e5fa2d\"/\"openshift-service-ca.crt\"" Oct 30 07:11:13.766207 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:13.766191 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e5fa2d\"/\"default-dockercfg-r2klx\"" Oct 30 07:11:13.766394 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:13.766378 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e5fa2d\"/\"distinct-w2-e5fa2d-kserve-self-signed-certs\"" Oct 30 07:11:13.772977 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:13.772943 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792"] Oct 30 07:11:13.847310 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:13.847271 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb929374-4eb9-4a81-b81a-b89098b9fb20-home\") pod \"distinct-w2-e5fa2d-kserve-b68f59c4b-df792\" (UID: \"bb929374-4eb9-4a81-b81a-b89098b9fb20\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:11:13.847512 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:13.847338 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb929374-4eb9-4a81-b81a-b89098b9fb20-tls-certs\") pod \"distinct-w2-e5fa2d-kserve-b68f59c4b-df792\" (UID: \"bb929374-4eb9-4a81-b81a-b89098b9fb20\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:11:13.847512 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:13.847433 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb929374-4eb9-4a81-b81a-b89098b9fb20-tmp-dir\") pod \"distinct-w2-e5fa2d-kserve-b68f59c4b-df792\" (UID: \"bb929374-4eb9-4a81-b81a-b89098b9fb20\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:11:13.847651 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:13.847525 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb929374-4eb9-4a81-b81a-b89098b9fb20-dshm\") pod \"distinct-w2-e5fa2d-kserve-b68f59c4b-df792\" (UID: \"bb929374-4eb9-4a81-b81a-b89098b9fb20\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:11:13.847651 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:13.847548 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc5hx\" (UniqueName: \"kubernetes.io/projected/bb929374-4eb9-4a81-b81a-b89098b9fb20-kube-api-access-sc5hx\") pod \"distinct-w2-e5fa2d-kserve-b68f59c4b-df792\" (UID: \"bb929374-4eb9-4a81-b81a-b89098b9fb20\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:11:13.847651 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:13.847629 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb929374-4eb9-4a81-b81a-b89098b9fb20-model-cache\") pod \"distinct-w2-e5fa2d-kserve-b68f59c4b-df792\" (UID: \"bb929374-4eb9-4a81-b81a-b89098b9fb20\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:11:13.958393 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:13.958337 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb929374-4eb9-4a81-b81a-b89098b9fb20-home\") pod \"distinct-w2-e5fa2d-kserve-b68f59c4b-df792\" (UID: \"bb929374-4eb9-4a81-b81a-b89098b9fb20\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:11:13.958801 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:13.958704 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb929374-4eb9-4a81-b81a-b89098b9fb20-tls-certs\") pod \"distinct-w2-e5fa2d-kserve-b68f59c4b-df792\" (UID: \"bb929374-4eb9-4a81-b81a-b89098b9fb20\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:11:13.958981 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:13.958962 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb929374-4eb9-4a81-b81a-b89098b9fb20-tmp-dir\") pod \"distinct-w2-e5fa2d-kserve-b68f59c4b-df792\" (UID: \"bb929374-4eb9-4a81-b81a-b89098b9fb20\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:11:13.959175 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:13.959132 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb929374-4eb9-4a81-b81a-b89098b9fb20-dshm\") pod \"distinct-w2-e5fa2d-kserve-b68f59c4b-df792\" (UID: \"bb929374-4eb9-4a81-b81a-b89098b9fb20\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:11:13.959293 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:13.959274 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc5hx\" (UniqueName: \"kubernetes.io/projected/bb929374-4eb9-4a81-b81a-b89098b9fb20-kube-api-access-sc5hx\") pod \"distinct-w2-e5fa2d-kserve-b68f59c4b-df792\" (UID: \"bb929374-4eb9-4a81-b81a-b89098b9fb20\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:11:13.970626 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:13.959448 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb929374-4eb9-4a81-b81a-b89098b9fb20-model-cache\") pod \"distinct-w2-e5fa2d-kserve-b68f59c4b-df792\" (UID: \"bb929374-4eb9-4a81-b81a-b89098b9fb20\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:11:13.971738 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:13.965034 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb929374-4eb9-4a81-b81a-b89098b9fb20-model-cache\") pod \"distinct-w2-e5fa2d-kserve-b68f59c4b-df792\" (UID: \"bb929374-4eb9-4a81-b81a-b89098b9fb20\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:11:13.971872 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:13.965265 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb929374-4eb9-4a81-b81a-b89098b9fb20-home\") pod \"distinct-w2-e5fa2d-kserve-b68f59c4b-df792\" (UID: \"bb929374-4eb9-4a81-b81a-b89098b9fb20\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:11:13.971872 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:13.971239 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb929374-4eb9-4a81-b81a-b89098b9fb20-tmp-dir\") pod \"distinct-w2-e5fa2d-kserve-b68f59c4b-df792\" (UID: \"bb929374-4eb9-4a81-b81a-b89098b9fb20\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:11:13.974597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:13.974544 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb929374-4eb9-4a81-b81a-b89098b9fb20-tls-certs\") pod \"distinct-w2-e5fa2d-kserve-b68f59c4b-df792\" (UID: \"bb929374-4eb9-4a81-b81a-b89098b9fb20\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:11:13.974765 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:13.974673 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb929374-4eb9-4a81-b81a-b89098b9fb20-dshm\") pod \"distinct-w2-e5fa2d-kserve-b68f59c4b-df792\" (UID: \"bb929374-4eb9-4a81-b81a-b89098b9fb20\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:11:13.993883 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:13.993803 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc5hx\" (UniqueName: \"kubernetes.io/projected/bb929374-4eb9-4a81-b81a-b89098b9fb20-kube-api-access-sc5hx\") pod \"distinct-w2-e5fa2d-kserve-b68f59c4b-df792\" (UID: \"bb929374-4eb9-4a81-b81a-b89098b9fb20\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:11:14.055005 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:14.054968 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:11:14.357739 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:14.357712 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792"] Oct 30 07:11:14.359592 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:11:14.359551 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb929374_4eb9_4a81_b81a_b89098b9fb20.slice/crio-6b656b7702b589391d33edd3778363fd2c04b0d4749a4992346c94a75ad5c0da WatchSource:0}: Error finding container 6b656b7702b589391d33edd3778363fd2c04b0d4749a4992346c94a75ad5c0da: Status 404 returned error can't find the container with id 6b656b7702b589391d33edd3778363fd2c04b0d4749a4992346c94a75ad5c0da Oct 30 07:11:14.431909 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:14.431869 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" event={"ID":"bb929374-4eb9-4a81-b81a-b89098b9fb20","Type":"ContainerStarted","Data":"6b656b7702b589391d33edd3778363fd2c04b0d4749a4992346c94a75ad5c0da"} Oct 30 07:11:15.440381 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:15.440345 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" event={"ID":"bb929374-4eb9-4a81-b81a-b89098b9fb20","Type":"ContainerStarted","Data":"c3076f46ec1ee0747238205ec259f639539225b459d68004da03f1a89ff3c791"} Oct 30 07:11:15.479664 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:15.479617 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podStartSLOduration=2.479599655 podStartE2EDuration="2.479599655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:11:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:11:15.479189509 +0000 UTC m=+881.822237498" watchObservedRunningTime="2026-10-30 07:11:15.479599655 +0000 UTC m=+881.822647622" Oct 30 07:11:16.055504 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:16.055466 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:11:23.022530 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:11:23.022492 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:11:24.055715 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:24.055673 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:11:24.074188 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:24.074156 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:11:24.469391 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:24.469304 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5"] Oct 30 07:11:24.474102 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:24.474074 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5" Oct 30 07:11:24.489881 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:24.489844 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a0b3f5\"/\"e2e-conflict-b-7c5e6b60-kserve-self-signed-certs\"" Oct 30 07:11:24.502881 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:24.502848 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5"] Oct 30 07:11:24.506516 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:24.506479 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:11:24.599713 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:24.599672 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fc2273c-b77e-404f-9df4-54257ed36565-home\") pod \"e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5\" (UID: \"0fc2273c-b77e-404f-9df4-54257ed36565\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5" Oct 30 07:11:24.599884 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:24.599768 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc2273c-b77e-404f-9df4-54257ed36565-tmp-dir\") pod \"e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5\" (UID: \"0fc2273c-b77e-404f-9df4-54257ed36565\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5" Oct 30 07:11:24.600016 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:24.599989 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jvmz\" (UniqueName: \"kubernetes.io/projected/0fc2273c-b77e-404f-9df4-54257ed36565-kube-api-access-2jvmz\") pod \"e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5\" (UID: \"0fc2273c-b77e-404f-9df4-54257ed36565\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5" Oct 30 07:11:24.600091 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:24.600065 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc2273c-b77e-404f-9df4-54257ed36565-tls-certs\") pod \"e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5\" (UID: \"0fc2273c-b77e-404f-9df4-54257ed36565\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5" Oct 30 07:11:24.600155 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:24.600093 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc2273c-b77e-404f-9df4-54257ed36565-dshm\") pod \"e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5\" (UID: \"0fc2273c-b77e-404f-9df4-54257ed36565\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5" Oct 30 07:11:24.600155 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:24.600118 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc2273c-b77e-404f-9df4-54257ed36565-model-cache\") pod \"e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5\" (UID: \"0fc2273c-b77e-404f-9df4-54257ed36565\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5" Oct 30 07:11:24.724656 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:24.709882 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jvmz\" (UniqueName: \"kubernetes.io/projected/0fc2273c-b77e-404f-9df4-54257ed36565-kube-api-access-2jvmz\") pod \"e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5\" (UID: \"0fc2273c-b77e-404f-9df4-54257ed36565\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5" Oct 30 07:11:24.724993 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:24.724957 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc2273c-b77e-404f-9df4-54257ed36565-tls-certs\") pod \"e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5\" (UID: \"0fc2273c-b77e-404f-9df4-54257ed36565\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5" Oct 30 07:11:24.725156 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:24.725140 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc2273c-b77e-404f-9df4-54257ed36565-dshm\") pod \"e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5\" (UID: \"0fc2273c-b77e-404f-9df4-54257ed36565\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5" Oct 30 07:11:24.725269 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:24.725255 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc2273c-b77e-404f-9df4-54257ed36565-model-cache\") pod \"e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5\" (UID: \"0fc2273c-b77e-404f-9df4-54257ed36565\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5" Oct 30 07:11:24.725417 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:24.725405 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fc2273c-b77e-404f-9df4-54257ed36565-home\") pod \"e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5\" (UID: \"0fc2273c-b77e-404f-9df4-54257ed36565\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5" Oct 30 07:11:24.725583 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:24.725556 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc2273c-b77e-404f-9df4-54257ed36565-tmp-dir\") pod \"e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5\" (UID: \"0fc2273c-b77e-404f-9df4-54257ed36565\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5" Oct 30 07:11:24.725994 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:24.725961 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc2273c-b77e-404f-9df4-54257ed36565-model-cache\") pod \"e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5\" (UID: \"0fc2273c-b77e-404f-9df4-54257ed36565\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5" Oct 30 07:11:24.726325 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:24.726302 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc2273c-b77e-404f-9df4-54257ed36565-tmp-dir\") pod \"e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5\" (UID: \"0fc2273c-b77e-404f-9df4-54257ed36565\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5" Oct 30 07:11:24.726726 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:24.726705 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fc2273c-b77e-404f-9df4-54257ed36565-home\") pod \"e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5\" (UID: \"0fc2273c-b77e-404f-9df4-54257ed36565\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5" Oct 30 07:11:24.742829 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:24.742563 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jvmz\" (UniqueName: \"kubernetes.io/projected/0fc2273c-b77e-404f-9df4-54257ed36565-kube-api-access-2jvmz\") pod \"e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5\" (UID: \"0fc2273c-b77e-404f-9df4-54257ed36565\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5" Oct 30 07:11:24.746058 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:24.746027 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc2273c-b77e-404f-9df4-54257ed36565-dshm\") pod \"e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5\" (UID: \"0fc2273c-b77e-404f-9df4-54257ed36565\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5" Oct 30 07:11:24.746273 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:24.746077 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc2273c-b77e-404f-9df4-54257ed36565-tls-certs\") pod \"e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5\" (UID: \"0fc2273c-b77e-404f-9df4-54257ed36565\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5" Oct 30 07:11:24.789152 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:24.789105 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5" Oct 30 07:11:25.093463 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:25.093431 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5"] Oct 30 07:11:25.095028 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:11:25.094993 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fc2273c_b77e_404f_9df4_54257ed36565.slice/crio-c0e87474e755b3ab6c58dbb308c525ea24913170ee85df21f7c21b595f8056d9 WatchSource:0}: Error finding container c0e87474e755b3ab6c58dbb308c525ea24913170ee85df21f7c21b595f8056d9: Status 404 returned error can't find the container with id c0e87474e755b3ab6c58dbb308c525ea24913170ee85df21f7c21b595f8056d9 Oct 30 07:11:25.503415 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:25.503380 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5" event={"ID":"0fc2273c-b77e-404f-9df4-54257ed36565","Type":"ContainerStarted","Data":"fbccfb3d131fb663334d9f23dbd3a8a0ac83b3dac7fdff98565cfc8e7748892c"} Oct 30 07:11:25.503609 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:25.503429 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5" event={"ID":"0fc2273c-b77e-404f-9df4-54257ed36565","Type":"ContainerStarted","Data":"c0e87474e755b3ab6c58dbb308c525ea24913170ee85df21f7c21b595f8056d9"} Oct 30 07:11:25.549086 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:25.549024 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5" podStartSLOduration=1.549005297 podStartE2EDuration="1.549005297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:11:25.546538125 +0000 UTC m=+891.889586089" watchObservedRunningTime="2026-10-30 07:11:25.549005297 +0000 UTC m=+891.892053263" Oct 30 07:11:25.789691 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:25.789618 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5" Oct 30 07:11:28.786618 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:28.786567 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs"] Oct 30 07:11:28.790767 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:28.790742 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:11:28.819364 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:28.819338 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs"] Oct 30 07:11:28.882324 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:28.882293 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/510f006a-0511-4993-8fdc-9e5ee1d6f048-istio-token\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:11:28.882502 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:28.882426 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-workload-certs\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:11:28.882561 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:28.882493 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-istio-envoy\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:11:28.882625 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:28.882564 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-workload-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:11:28.882625 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:28.882607 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/510f006a-0511-4993-8fdc-9e5ee1d6f048-istiod-ca-cert\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:11:28.882721 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:28.882699 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-istio-data\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:11:28.882821 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:28.882801 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8td8h\" (UniqueName: \"kubernetes.io/projected/510f006a-0511-4993-8fdc-9e5ee1d6f048-kube-api-access-8td8h\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:11:28.882855 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:28.882841 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/510f006a-0511-4993-8fdc-9e5ee1d6f048-istio-podinfo\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:11:28.882914 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:28.882903 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-credential-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:11:28.984100 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:28.984056 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-workload-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:11:28.984333 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:28.984111 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/510f006a-0511-4993-8fdc-9e5ee1d6f048-istiod-ca-cert\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:11:28.984333 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:28.984170 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-istio-data\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:11:28.984333 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:28.984245 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8td8h\" (UniqueName: \"kubernetes.io/projected/510f006a-0511-4993-8fdc-9e5ee1d6f048-kube-api-access-8td8h\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:11:28.984333 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:28.984283 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/510f006a-0511-4993-8fdc-9e5ee1d6f048-istio-podinfo\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:11:28.984333 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:28.984328 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-credential-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:11:28.984617 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:28.984357 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/510f006a-0511-4993-8fdc-9e5ee1d6f048-istio-token\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:11:28.984617 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:28.984444 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-workload-certs\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:11:28.984617 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:28.984469 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-istio-envoy\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:11:28.985606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:28.985530 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-workload-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:11:28.985945 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:28.985777 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/510f006a-0511-4993-8fdc-9e5ee1d6f048-istiod-ca-cert\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:11:28.985945 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:28.985903 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-workload-certs\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:11:28.986147 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:28.986120 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-credential-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:11:28.986258 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:28.986245 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-istio-data\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:11:28.987107 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:28.987083 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-istio-envoy\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:11:28.988199 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:28.988171 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/510f006a-0511-4993-8fdc-9e5ee1d6f048-istio-podinfo\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:11:29.007284 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:29.007249 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/510f006a-0511-4993-8fdc-9e5ee1d6f048-istio-token\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:11:29.007513 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:29.007491 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8td8h\" (UniqueName: \"kubernetes.io/projected/510f006a-0511-4993-8fdc-9e5ee1d6f048-kube-api-access-8td8h\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:11:29.105188 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:29.105093 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:11:29.435956 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:29.435918 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs"] Oct 30 07:11:29.437195 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:11:29.437163 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510f006a_0511_4993_8fdc_9e5ee1d6f048.slice/crio-5e60a7fca814722342bac51199fe72c136fc3fe4c02a520fb3ae9ea586bcc781 WatchSource:0}: Error finding container 5e60a7fca814722342bac51199fe72c136fc3fe4c02a520fb3ae9ea586bcc781: Status 404 returned error can't find the container with id 5e60a7fca814722342bac51199fe72c136fc3fe4c02a520fb3ae9ea586bcc781 Oct 30 07:11:29.439778 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:29.439748 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:11:29.439848 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:29.439810 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:11:29.439848 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:29.439840 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:11:29.531829 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:29.531784 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" event={"ID":"510f006a-0511-4993-8fdc-9e5ee1d6f048","Type":"ContainerStarted","Data":"9a966c0166ca4ddcf117e4671e93a491478852d7592fccdbe9ed7185f51d2532"} Oct 30 07:11:29.531994 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:29.531851 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" event={"ID":"510f006a-0511-4993-8fdc-9e5ee1d6f048","Type":"ContainerStarted","Data":"5e60a7fca814722342bac51199fe72c136fc3fe4c02a520fb3ae9ea586bcc781"} Oct 30 07:11:29.579302 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:29.579227 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" podStartSLOduration=1.579207059 podStartE2EDuration="1.579207059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:11:29.57653633 +0000 UTC m=+895.919584322" watchObservedRunningTime="2026-10-30 07:11:29.579207059 +0000 UTC m=+895.922255024" Oct 30 07:11:30.105543 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:30.105504 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:11:30.107470 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:30.107444 4951 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 07:11:30.107600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:30.107495 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" podUID="510f006a-0511-4993-8fdc-9e5ee1d6f048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 07:11:31.106154 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:31.106121 4951 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 07:11:31.106563 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:31.106175 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" podUID="510f006a-0511-4993-8fdc-9e5ee1d6f048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 07:11:32.106356 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:32.106316 4951 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 07:11:32.106912 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:32.106411 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" podUID="510f006a-0511-4993-8fdc-9e5ee1d6f048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 07:11:33.105955 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:33.105917 4951 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 07:11:33.106117 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:33.105977 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" podUID="510f006a-0511-4993-8fdc-9e5ee1d6f048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 07:11:33.992598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:33.992552 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/3.log" Oct 30 07:11:33.999600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:33.999549 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/3.log" Oct 30 07:11:34.002048 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:34.002032 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 07:11:34.007938 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:34.007916 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 07:11:34.105913 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:34.105881 4951 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 07:11:34.106089 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:34.105937 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" podUID="510f006a-0511-4993-8fdc-9e5ee1d6f048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 07:11:34.791010 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:34.790973 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5" Oct 30 07:11:34.807500 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:34.807462 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5" Oct 30 07:11:35.106267 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:35.106178 4951 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 07:11:35.106267 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:35.106235 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" podUID="510f006a-0511-4993-8fdc-9e5ee1d6f048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 07:11:35.618972 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:35.618941 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5" Oct 30 07:11:36.110445 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:36.110415 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:11:36.110969 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:36.110830 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:11:36.111627 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:36.111605 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:11:37.027893 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.027849 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8"] Oct 30 07:11:37.033601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.033564 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:11:37.054300 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.054279 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-989104-openshift-default-dockercfg-59vg7\"" Oct 30 07:11:37.062746 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.062717 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8"] Oct 30 07:11:37.208120 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.208082 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:11:37.208601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.208136 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012bfdea-4eb7-442d-a951-d1fd23882318-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:11:37.208601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.208167 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012bfdea-4eb7-442d-a951-d1fd23882318-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:11:37.208601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.208300 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:11:37.208601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.208409 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:11:37.208601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.208484 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:11:37.208601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.208566 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012bfdea-4eb7-442d-a951-d1fd23882318-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:11:37.208860 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.208659 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2wcm\" (UniqueName: \"kubernetes.io/projected/012bfdea-4eb7-442d-a951-d1fd23882318-kube-api-access-f2wcm\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:11:37.208860 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.208681 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:11:37.309354 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.309263 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2wcm\" (UniqueName: \"kubernetes.io/projected/012bfdea-4eb7-442d-a951-d1fd23882318-kube-api-access-f2wcm\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:11:37.309354 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.309302 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:11:37.309354 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.309352 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:11:37.309680 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.309397 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012bfdea-4eb7-442d-a951-d1fd23882318-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:11:37.309680 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.309435 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012bfdea-4eb7-442d-a951-d1fd23882318-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:11:37.309680 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.309600 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:11:37.309843 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.309703 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:11:37.309843 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.309784 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:11:37.309843 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.309823 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:11:37.310006 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.309851 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012bfdea-4eb7-442d-a951-d1fd23882318-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:11:37.310119 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.310099 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:11:37.310189 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.310159 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012bfdea-4eb7-442d-a951-d1fd23882318-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:11:37.310324 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.310301 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:11:37.310410 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.310345 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:11:37.311737 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.311710 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:11:37.312033 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.312016 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012bfdea-4eb7-442d-a951-d1fd23882318-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:11:37.331825 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.331794 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012bfdea-4eb7-442d-a951-d1fd23882318-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:11:37.332143 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.332121 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2wcm\" (UniqueName: \"kubernetes.io/projected/012bfdea-4eb7-442d-a951-d1fd23882318-kube-api-access-f2wcm\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:11:37.347793 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.347772 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:11:37.666981 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.666954 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8"] Oct 30 07:11:37.668452 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:11:37.668426 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod012bfdea_4eb7_442d_a951_d1fd23882318.slice/crio-fcc3aba3d885e0c87bee1764dbce26399a078e506f486ada4638ea0513db0164 WatchSource:0}: Error finding container fcc3aba3d885e0c87bee1764dbce26399a078e506f486ada4638ea0513db0164: Status 404 returned error can't find the container with id fcc3aba3d885e0c87bee1764dbce26399a078e506f486ada4638ea0513db0164 Oct 30 07:11:37.671217 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.671185 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:11:37.671321 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.671250 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:11:37.671321 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:37.671280 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:11:38.646838 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:38.646792 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" event={"ID":"012bfdea-4eb7-442d-a951-d1fd23882318","Type":"ContainerStarted","Data":"8c0a7b6f94a9b8c382354be1b87553e07f84fe070e21a8c795e58aef60fe5ac0"} Oct 30 07:11:38.647809 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:38.647777 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" event={"ID":"012bfdea-4eb7-442d-a951-d1fd23882318","Type":"ContainerStarted","Data":"fcc3aba3d885e0c87bee1764dbce26399a078e506f486ada4638ea0513db0164"} Oct 30 07:11:38.717808 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:38.717756 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" podStartSLOduration=2.7177423000000003 podStartE2EDuration="2.7177423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:11:38.705362687 +0000 UTC m=+905.048410693" watchObservedRunningTime="2026-10-30 07:11:38.7177423 +0000 UTC m=+905.060790265" Oct 30 07:11:39.351321 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:39.349938 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:11:39.353873 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:39.353706 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 07:11:39.353873 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:39.353799 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" podUID="012bfdea-4eb7-442d-a951-d1fd23882318" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 07:11:40.349073 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:40.349028 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 07:11:40.349764 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:40.349718 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" podUID="012bfdea-4eb7-442d-a951-d1fd23882318" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 07:11:41.351602 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:41.350942 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 07:11:41.351602 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:41.351027 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" podUID="012bfdea-4eb7-442d-a951-d1fd23882318" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 07:11:42.348975 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:42.348935 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 07:11:42.349249 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:42.349217 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" podUID="012bfdea-4eb7-442d-a951-d1fd23882318" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 07:11:43.350000 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:43.349963 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 07:11:43.350409 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:43.350044 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" podUID="012bfdea-4eb7-442d-a951-d1fd23882318" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 07:11:44.154224 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.154187 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577"] Oct 30 07:11:44.158297 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.158276 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:11:44.183335 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.183299 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577"] Oct 30 07:11:44.227961 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.227919 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dq2c\" (UniqueName: \"kubernetes.io/projected/5c9605f5-3aa1-4f48-8ec6-1b709863952f-kube-api-access-5dq2c\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:11:44.228108 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.227976 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:11:44.228108 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.228055 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:11:44.228108 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.228079 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:11:44.228264 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.228147 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:11:44.228264 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.228192 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:11:44.228264 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.228221 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:11:44.228400 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.228261 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:11:44.228449 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.228391 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:11:44.329724 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.329678 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:11:44.329920 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.329796 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dq2c\" (UniqueName: \"kubernetes.io/projected/5c9605f5-3aa1-4f48-8ec6-1b709863952f-kube-api-access-5dq2c\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:11:44.329920 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.329829 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:11:44.329920 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.329906 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:11:44.330099 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.330019 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:11:44.330152 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.330107 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:11:44.330227 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.330156 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:11:44.330227 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.330192 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:11:44.330327 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.330233 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:11:44.330467 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.330443 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:11:44.330845 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.330820 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:11:44.331008 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.330890 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:11:44.331249 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.331219 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:11:44.331249 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.331240 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:11:44.333166 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.333140 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:11:44.333620 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.333588 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:11:44.352499 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.352467 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:11:44.352992 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.352966 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dq2c\" (UniqueName: \"kubernetes.io/projected/5c9605f5-3aa1-4f48-8ec6-1b709863952f-kube-api-access-5dq2c\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:11:44.478056 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.477975 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:11:44.745792 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.745714 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:11:44.746705 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.746681 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:11:44.747390 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.747371 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:11:44.800002 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.799973 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577"] Oct 30 07:11:44.801621 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:11:44.801593 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c9605f5_3aa1_4f48_8ec6_1b709863952f.slice/crio-ecff96ffaa1444a2e50773d0c10217e3f3806376ca37e78c0e43a20163ed05da WatchSource:0}: Error finding container ecff96ffaa1444a2e50773d0c10217e3f3806376ca37e78c0e43a20163ed05da: Status 404 returned error can't find the container with id ecff96ffaa1444a2e50773d0c10217e3f3806376ca37e78c0e43a20163ed05da Oct 30 07:11:44.804015 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.803984 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:11:44.804116 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.804057 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:11:44.804116 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:44.804104 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:11:45.702364 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:45.702325 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" event={"ID":"5c9605f5-3aa1-4f48-8ec6-1b709863952f","Type":"ContainerStarted","Data":"1040c97709b625ce4ea3ceb35d515970bc078dc34ac185cb993bff353c1eb58f"} Oct 30 07:11:45.702875 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:45.702391 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" event={"ID":"5c9605f5-3aa1-4f48-8ec6-1b709863952f","Type":"ContainerStarted","Data":"ecff96ffaa1444a2e50773d0c10217e3f3806376ca37e78c0e43a20163ed05da"} Oct 30 07:11:45.773462 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:45.773409 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" podStartSLOduration=1.773396076 podStartE2EDuration="1.773396076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:11:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:11:45.762392933 +0000 UTC m=+912.105440922" watchObservedRunningTime="2026-10-30 07:11:45.773396076 +0000 UTC m=+912.116444041" Oct 30 07:11:46.478554 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:46.478516 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:11:46.480167 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:46.480141 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 07:11:46.480282 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:46.480189 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" podUID="5c9605f5-3aa1-4f48-8ec6-1b709863952f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 07:11:47.478374 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:47.478330 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 07:11:47.478787 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:47.478395 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" podUID="5c9605f5-3aa1-4f48-8ec6-1b709863952f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 07:11:48.479059 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:48.479017 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 07:11:48.479491 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:48.479079 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" podUID="5c9605f5-3aa1-4f48-8ec6-1b709863952f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 07:11:49.479122 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:49.479084 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 07:11:49.479539 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:49.479149 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" podUID="5c9605f5-3aa1-4f48-8ec6-1b709863952f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 07:11:50.478309 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:50.478274 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 07:11:50.478483 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:50.478327 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" podUID="5c9605f5-3aa1-4f48-8ec6-1b709863952f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 07:11:52.033597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:52.033100 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:11:52.034122 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:52.033997 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:11:52.039595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:11:52.036229 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:11:53.024541 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:11:53.024502 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:12:01.258955 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:01.258904 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5"] Oct 30 07:12:13.949567 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:13.949528 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv"] Oct 30 07:12:13.955293 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:13.955273 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:12:13.984147 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:13.984106 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-credential-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:12:13.984319 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:13.984248 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a97844ee-24e2-4039-80d1-559b7b9f03bd-istiod-ca-cert\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:12:13.984319 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:13.984286 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-istio-data\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:12:13.984428 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:13.984326 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a97844ee-24e2-4039-80d1-559b7b9f03bd-istio-token\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:12:13.984428 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:13.984351 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kppmq\" (UniqueName: \"kubernetes.io/projected/a97844ee-24e2-4039-80d1-559b7b9f03bd-kube-api-access-kppmq\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:12:13.984428 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:13.984385 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a97844ee-24e2-4039-80d1-559b7b9f03bd-istio-podinfo\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:12:13.984595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:13.984427 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-workload-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:12:13.984595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:13.984496 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-istio-envoy\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:12:13.984595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:13.984526 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-workload-certs\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:12:13.988368 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:13.988334 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv"] Oct 30 07:12:14.086102 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:14.086057 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a97844ee-24e2-4039-80d1-559b7b9f03bd-istio-podinfo\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:12:14.086304 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:14.086118 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-workload-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:12:14.086304 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:14.086176 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-istio-envoy\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:12:14.086304 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:14.086201 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-workload-certs\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:12:14.086304 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:14.086256 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-credential-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:12:14.086517 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:14.086380 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a97844ee-24e2-4039-80d1-559b7b9f03bd-istiod-ca-cert\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:12:14.086517 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:14.086416 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-istio-data\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:12:14.086517 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:14.086456 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a97844ee-24e2-4039-80d1-559b7b9f03bd-istio-token\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:12:14.086517 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:14.086482 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kppmq\" (UniqueName: \"kubernetes.io/projected/a97844ee-24e2-4039-80d1-559b7b9f03bd-kube-api-access-kppmq\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:12:14.086751 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:14.086676 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-workload-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:12:14.086853 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:14.086827 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-credential-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:12:14.087103 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:14.087081 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-workload-certs\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:12:14.087180 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:14.087097 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a97844ee-24e2-4039-80d1-559b7b9f03bd-istiod-ca-cert\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:12:14.087180 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:14.087147 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-istio-data\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:12:14.088861 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:14.088834 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a97844ee-24e2-4039-80d1-559b7b9f03bd-istio-podinfo\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:12:14.088978 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:14.088907 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-istio-envoy\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:12:14.108684 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:14.108647 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a97844ee-24e2-4039-80d1-559b7b9f03bd-istio-token\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:12:14.109029 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:14.109004 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kppmq\" (UniqueName: \"kubernetes.io/projected/a97844ee-24e2-4039-80d1-559b7b9f03bd-kube-api-access-kppmq\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:12:14.272474 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:14.272441 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:12:14.613728 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:14.613702 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv"] Oct 30 07:12:14.615798 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:12:14.615762 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda97844ee_24e2_4039_80d1_559b7b9f03bd.slice/crio-3cd3314d02c6fec1bb748399916e1e8f5313e9731c2700e63b4c6ce8888a0868 WatchSource:0}: Error finding container 3cd3314d02c6fec1bb748399916e1e8f5313e9731c2700e63b4c6ce8888a0868: Status 404 returned error can't find the container with id 3cd3314d02c6fec1bb748399916e1e8f5313e9731c2700e63b4c6ce8888a0868 Oct 30 07:12:14.618289 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:14.618256 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:12:14.618376 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:14.618326 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:12:14.618376 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:14.618358 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:12:14.906972 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:14.906873 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" event={"ID":"a97844ee-24e2-4039-80d1-559b7b9f03bd","Type":"ContainerStarted","Data":"5cf2f8c236380edc6c9a9ed7ecf088dc7cae92e333ecbc9a43f8f14db252293b"} Oct 30 07:12:14.906972 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:14.906938 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" event={"ID":"a97844ee-24e2-4039-80d1-559b7b9f03bd","Type":"ContainerStarted","Data":"3cd3314d02c6fec1bb748399916e1e8f5313e9731c2700e63b4c6ce8888a0868"} Oct 30 07:12:14.953546 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:14.953488 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" podStartSLOduration=1.953474239 podStartE2EDuration="1.953474239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:14.950186484 +0000 UTC m=+941.293234449" watchObservedRunningTime="2026-10-30 07:12:14.953474239 +0000 UTC m=+941.296522203" Oct 30 07:12:15.272998 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:15.272963 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:12:15.274599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:15.274559 4951 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 07:12:15.274734 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:15.274635 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" podUID="a97844ee-24e2-4039-80d1-559b7b9f03bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 07:12:16.273736 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:16.273706 4951 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 07:12:16.274125 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:16.273755 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" podUID="a97844ee-24e2-4039-80d1-559b7b9f03bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 07:12:16.303119 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:16.303067 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5" podUID="0fc2273c-b77e-404f-9df4-54257ed36565" containerName="main" containerID="cri-o://fbccfb3d131fb663334d9f23dbd3a8a0ac83b3dac7fdff98565cfc8e7748892c" gracePeriod=45 Oct 30 07:12:16.652636 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:16.652604 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5" Oct 30 07:12:16.721220 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:16.721184 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fc2273c-b77e-404f-9df4-54257ed36565-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc2273c-b77e-404f-9df4-54257ed36565-dshm\") pod \"0fc2273c-b77e-404f-9df4-54257ed36565\" (UID: \"0fc2273c-b77e-404f-9df4-54257ed36565\") " Oct 30 07:12:16.721420 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:16.721238 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fc2273c-b77e-404f-9df4-54257ed36565-home\" (UniqueName: \"kubernetes.io/empty-dir/0fc2273c-b77e-404f-9df4-54257ed36565-home\") pod \"0fc2273c-b77e-404f-9df4-54257ed36565\" (UID: \"0fc2273c-b77e-404f-9df4-54257ed36565\") " Oct 30 07:12:16.721420 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:16.721332 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fc2273c-b77e-404f-9df4-54257ed36565-kube-api-access-2jvmz\" (UniqueName: \"kubernetes.io/projected/0fc2273c-b77e-404f-9df4-54257ed36565-kube-api-access-2jvmz\") pod \"0fc2273c-b77e-404f-9df4-54257ed36565\" (UID: \"0fc2273c-b77e-404f-9df4-54257ed36565\") " Oct 30 07:12:16.721420 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:16.721365 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fc2273c-b77e-404f-9df4-54257ed36565-tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc2273c-b77e-404f-9df4-54257ed36565-tls-certs\") pod \"0fc2273c-b77e-404f-9df4-54257ed36565\" (UID: \"0fc2273c-b77e-404f-9df4-54257ed36565\") " Oct 30 07:12:16.721658 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:16.721427 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fc2273c-b77e-404f-9df4-54257ed36565-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc2273c-b77e-404f-9df4-54257ed36565-model-cache\") pod \"0fc2273c-b77e-404f-9df4-54257ed36565\" (UID: \"0fc2273c-b77e-404f-9df4-54257ed36565\") " Oct 30 07:12:16.721658 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:16.721507 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fc2273c-b77e-404f-9df4-54257ed36565-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc2273c-b77e-404f-9df4-54257ed36565-tmp-dir\") pod \"0fc2273c-b77e-404f-9df4-54257ed36565\" (UID: \"0fc2273c-b77e-404f-9df4-54257ed36565\") " Oct 30 07:12:16.721658 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:16.721556 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc2273c-b77e-404f-9df4-54257ed36565-home" pod "0fc2273c-b77e-404f-9df4-54257ed36565" (UID: "0fc2273c-b77e-404f-9df4-54257ed36565"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:12:16.753624 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:16.722052 4951 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fc2273c-b77e-404f-9df4-54257ed36565-home\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:12:16.753624 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:16.727677 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc2273c-b77e-404f-9df4-54257ed36565-model-cache" pod "0fc2273c-b77e-404f-9df4-54257ed36565" (UID: "0fc2273c-b77e-404f-9df4-54257ed36565"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:12:16.753624 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:16.740672 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc2273c-b77e-404f-9df4-54257ed36565-tmp-dir" pod "0fc2273c-b77e-404f-9df4-54257ed36565" (UID: "0fc2273c-b77e-404f-9df4-54257ed36565"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:12:16.756942 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:16.756894 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fc2273c-b77e-404f-9df4-54257ed36565-tls-certs" pod "0fc2273c-b77e-404f-9df4-54257ed36565" (UID: "0fc2273c-b77e-404f-9df4-54257ed36565"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:12:16.758004 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:16.757974 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc2273c-b77e-404f-9df4-54257ed36565-kube-api-access-2jvmz" pod "0fc2273c-b77e-404f-9df4-54257ed36565" (UID: "0fc2273c-b77e-404f-9df4-54257ed36565"). InnerVolumeSpecName "kube-api-access-2jvmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:12:16.760995 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:16.760958 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc2273c-b77e-404f-9df4-54257ed36565-dshm" pod "0fc2273c-b77e-404f-9df4-54257ed36565" (UID: "0fc2273c-b77e-404f-9df4-54257ed36565"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:12:16.831600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:16.830662 4951 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc2273c-b77e-404f-9df4-54257ed36565-dshm\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:12:16.831600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:16.830702 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jvmz\" (UniqueName: \"kubernetes.io/projected/0fc2273c-b77e-404f-9df4-54257ed36565-kube-api-access-2jvmz\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:12:16.831600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:16.830720 4951 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc2273c-b77e-404f-9df4-54257ed36565-tls-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:12:16.831600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:16.830735 4951 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc2273c-b77e-404f-9df4-54257ed36565-model-cache\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:12:16.831600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:16.830750 4951 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc2273c-b77e-404f-9df4-54257ed36565-tmp-dir\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:12:16.919313 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:16.919214 4951 generic.go:356] "Generic (PLEG): container finished" podID="0fc2273c-b77e-404f-9df4-54257ed36565" containerID="fbccfb3d131fb663334d9f23dbd3a8a0ac83b3dac7fdff98565cfc8e7748892c" exitCode=0 Oct 30 07:12:16.919313 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:16.919269 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5" Oct 30 07:12:16.919313 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:16.919301 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5" event={"ID":"0fc2273c-b77e-404f-9df4-54257ed36565","Type":"ContainerDied","Data":"fbccfb3d131fb663334d9f23dbd3a8a0ac83b3dac7fdff98565cfc8e7748892c"} Oct 30 07:12:16.919614 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:16.919347 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5" event={"ID":"0fc2273c-b77e-404f-9df4-54257ed36565","Type":"ContainerDied","Data":"c0e87474e755b3ab6c58dbb308c525ea24913170ee85df21f7c21b595f8056d9"} Oct 30 07:12:16.919614 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:16.919361 4951 scope.go:122] "RemoveContainer" containerID="fbccfb3d131fb663334d9f23dbd3a8a0ac83b3dac7fdff98565cfc8e7748892c" Oct 30 07:12:16.953707 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:16.953684 4951 scope.go:122] "RemoveContainer" containerID="fbccfb3d131fb663334d9f23dbd3a8a0ac83b3dac7fdff98565cfc8e7748892c" Oct 30 07:12:16.954029 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:12:16.954009 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbccfb3d131fb663334d9f23dbd3a8a0ac83b3dac7fdff98565cfc8e7748892c\": container with ID starting with fbccfb3d131fb663334d9f23dbd3a8a0ac83b3dac7fdff98565cfc8e7748892c not found: ID does not exist" containerID="fbccfb3d131fb663334d9f23dbd3a8a0ac83b3dac7fdff98565cfc8e7748892c" Oct 30 07:12:16.954116 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:16.954040 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbccfb3d131fb663334d9f23dbd3a8a0ac83b3dac7fdff98565cfc8e7748892c"} err="failed to get container status \"fbccfb3d131fb663334d9f23dbd3a8a0ac83b3dac7fdff98565cfc8e7748892c\": rpc error: code = NotFound desc = could not find container \"fbccfb3d131fb663334d9f23dbd3a8a0ac83b3dac7fdff98565cfc8e7748892c\": container with ID starting with fbccfb3d131fb663334d9f23dbd3a8a0ac83b3dac7fdff98565cfc8e7748892c not found: ID does not exist" Oct 30 07:12:16.988281 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:16.988245 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5"] Oct 30 07:12:16.994691 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:16.994649 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5"] Oct 30 07:12:17.273623 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:17.273553 4951 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 07:12:17.273807 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:17.273648 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" podUID="a97844ee-24e2-4039-80d1-559b7b9f03bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 07:12:17.608025 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:17.607922 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-b-7c5e6b60-kserve-8776798c5-hzjp5" podUID="0fc2273c-b77e-404f-9df4-54257ed36565" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 07:12:17.771673 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:17.771635 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fc2273c-b77e-404f-9df4-54257ed36565" path="/var/lib/kubelet/pods/0fc2273c-b77e-404f-9df4-54257ed36565/volumes" Oct 30 07:12:18.272904 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:18.272858 4951 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 07:12:18.273116 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:18.272929 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" podUID="a97844ee-24e2-4039-80d1-559b7b9f03bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 07:12:19.273688 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:19.273652 4951 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 07:12:19.274221 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:19.273709 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" podUID="a97844ee-24e2-4039-80d1-559b7b9f03bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 07:12:20.050152 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:20.050115 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv"] Oct 30 07:12:20.055602 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:20.051246 4951 cpu_manager.go:414] "containerMap: removing container" podUID="0fc2273c-b77e-404f-9df4-54257ed36565" containerName="main" Oct 30 07:12:20.055844 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:20.055821 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fc2273c-b77e-404f-9df4-54257ed36565" containerName="main" Oct 30 07:12:20.056168 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:20.056150 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fc2273c-b77e-404f-9df4-54257ed36565" containerName="main" Oct 30 07:12:20.061705 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:20.061643 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:12:20.079009 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:20.078975 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv"] Oct 30 07:12:20.086970 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:20.086945 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b72d79\"/\"distinct-2-w0-b72d79-kserve-self-signed-certs\"" Oct 30 07:12:20.183503 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:20.183463 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-tls-certs\") pod \"distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv\" (UID: \"61d8d83a-32ea-43b0-9aa7-b70bff2217b8\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:12:20.183503 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:20.183509 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-model-cache\") pod \"distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv\" (UID: \"61d8d83a-32ea-43b0-9aa7-b70bff2217b8\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:12:20.183744 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:20.183717 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-tmp-dir\") pod \"distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv\" (UID: \"61d8d83a-32ea-43b0-9aa7-b70bff2217b8\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:12:20.183818 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:20.183803 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-dshm\") pod \"distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv\" (UID: \"61d8d83a-32ea-43b0-9aa7-b70bff2217b8\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:12:20.183908 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:20.183892 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n89sr\" (UniqueName: \"kubernetes.io/projected/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-kube-api-access-n89sr\") pod \"distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv\" (UID: \"61d8d83a-32ea-43b0-9aa7-b70bff2217b8\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:12:20.183998 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:20.183985 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-home\") pod \"distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv\" (UID: \"61d8d83a-32ea-43b0-9aa7-b70bff2217b8\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:12:20.281745 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:20.281711 4951 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 07:12:20.282356 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:20.282319 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" podUID="a97844ee-24e2-4039-80d1-559b7b9f03bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 07:12:20.292437 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:20.292398 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-tmp-dir\") pod \"distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv\" (UID: \"61d8d83a-32ea-43b0-9aa7-b70bff2217b8\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:12:20.293234 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:20.293207 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-dshm\") pod \"distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv\" (UID: \"61d8d83a-32ea-43b0-9aa7-b70bff2217b8\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:12:20.293459 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:20.293442 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n89sr\" (UniqueName: \"kubernetes.io/projected/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-kube-api-access-n89sr\") pod \"distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv\" (UID: \"61d8d83a-32ea-43b0-9aa7-b70bff2217b8\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:12:20.293621 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:20.293096 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-tmp-dir\") pod \"distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv\" (UID: \"61d8d83a-32ea-43b0-9aa7-b70bff2217b8\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:12:20.293800 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:20.293619 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-home\") pod \"distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv\" (UID: \"61d8d83a-32ea-43b0-9aa7-b70bff2217b8\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:12:20.293895 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:20.293876 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-home\") pod \"distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv\" (UID: \"61d8d83a-32ea-43b0-9aa7-b70bff2217b8\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:12:20.294099 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:20.294079 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-tls-certs\") pod \"distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv\" (UID: \"61d8d83a-32ea-43b0-9aa7-b70bff2217b8\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:12:20.294553 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:20.294534 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-model-cache\") pod \"distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv\" (UID: \"61d8d83a-32ea-43b0-9aa7-b70bff2217b8\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:12:20.295049 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:20.295029 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-model-cache\") pod \"distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv\" (UID: \"61d8d83a-32ea-43b0-9aa7-b70bff2217b8\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:12:20.324147 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:20.324071 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-dshm\") pod \"distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv\" (UID: \"61d8d83a-32ea-43b0-9aa7-b70bff2217b8\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:12:20.338598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:20.324650 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-tls-certs\") pod \"distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv\" (UID: \"61d8d83a-32ea-43b0-9aa7-b70bff2217b8\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:12:20.342594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:20.341053 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n89sr\" (UniqueName: \"kubernetes.io/projected/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-kube-api-access-n89sr\") pod \"distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv\" (UID: \"61d8d83a-32ea-43b0-9aa7-b70bff2217b8\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:12:20.391952 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:20.391726 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:12:20.721939 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:20.721912 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv"] Oct 30 07:12:20.723238 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:12:20.723208 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61d8d83a_32ea_43b0_9aa7_b70bff2217b8.slice/crio-af0fbaaaf46a9cd8fcc0c0f75b947f1e33745a3216331442604f03b589820977 WatchSource:0}: Error finding container af0fbaaaf46a9cd8fcc0c0f75b947f1e33745a3216331442604f03b589820977: Status 404 returned error can't find the container with id af0fbaaaf46a9cd8fcc0c0f75b947f1e33745a3216331442604f03b589820977 Oct 30 07:12:20.968945 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:20.968905 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" event={"ID":"61d8d83a-32ea-43b0-9aa7-b70bff2217b8","Type":"ContainerStarted","Data":"449c36e67c4891a77e26ef1ed03c3ca0eebfd10ef835103293fc97656d434e5e"} Oct 30 07:12:20.969150 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:20.968958 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" event={"ID":"61d8d83a-32ea-43b0-9aa7-b70bff2217b8","Type":"ContainerStarted","Data":"af0fbaaaf46a9cd8fcc0c0f75b947f1e33745a3216331442604f03b589820977"} Oct 30 07:12:21.010922 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:21.010880 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podStartSLOduration=1.010865986 podStartE2EDuration="1.010865986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:21.008663423 +0000 UTC m=+947.351711388" watchObservedRunningTime="2026-10-30 07:12:21.010865986 +0000 UTC m=+947.353913971" Oct 30 07:12:21.392513 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:21.392396 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:12:21.640011 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:21.639978 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:12:21.641505 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:21.641480 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:12:21.645293 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:21.644517 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:12:22.179707 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.179674 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m"] Oct 30 07:12:22.182830 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.182806 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:12:22.212022 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.211992 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m"] Oct 30 07:12:22.223133 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.223093 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:12:22.223320 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.223163 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5jpd\" (UniqueName: \"kubernetes.io/projected/4070bffc-3574-4ee1-b444-e405af814c82-kube-api-access-h5jpd\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:12:22.223320 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.223283 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4070bffc-3574-4ee1-b444-e405af814c82-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:12:22.223448 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.223364 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4070bffc-3574-4ee1-b444-e405af814c82-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:12:22.223514 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.223431 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:12:22.223514 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.223492 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:12:22.223661 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.223556 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:12:22.223661 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.223653 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:12:22.224023 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.223705 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4070bffc-3574-4ee1-b444-e405af814c82-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:12:22.324565 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.324524 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:12:22.324565 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.324586 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4070bffc-3574-4ee1-b444-e405af814c82-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:12:22.324850 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.324653 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:12:22.324850 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.324685 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5jpd\" (UniqueName: \"kubernetes.io/projected/4070bffc-3574-4ee1-b444-e405af814c82-kube-api-access-h5jpd\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:12:22.324850 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.324728 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4070bffc-3574-4ee1-b444-e405af814c82-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:12:22.324850 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.324786 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4070bffc-3574-4ee1-b444-e405af814c82-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:12:22.324850 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.324822 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:12:22.325130 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.325042 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:12:22.325385 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.324848 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:12:22.325483 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.325407 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:12:22.325483 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.325447 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:12:22.325769 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.325747 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:12:22.325906 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.325882 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:12:22.326040 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.325976 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4070bffc-3574-4ee1-b444-e405af814c82-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:12:22.327530 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.327509 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4070bffc-3574-4ee1-b444-e405af814c82-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:12:22.327940 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.327917 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:12:22.346621 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.346601 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4070bffc-3574-4ee1-b444-e405af814c82-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:12:22.348920 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.348900 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5jpd\" (UniqueName: \"kubernetes.io/projected/4070bffc-3574-4ee1-b444-e405af814c82-kube-api-access-h5jpd\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:12:22.500032 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.500003 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:12:22.812789 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.812758 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m"] Oct 30 07:12:22.814181 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:12:22.814137 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4070bffc_3574_4ee1_b444_e405af814c82.slice/crio-c1e6d08deb24261d1fe8ba5781b56e012344f5b73e91e19273e4a5eb885b0607 WatchSource:0}: Error finding container c1e6d08deb24261d1fe8ba5781b56e012344f5b73e91e19273e4a5eb885b0607: Status 404 returned error can't find the container with id c1e6d08deb24261d1fe8ba5781b56e012344f5b73e91e19273e4a5eb885b0607 Oct 30 07:12:22.816794 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.816760 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:12:22.816923 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.816838 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:12:22.816923 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.816866 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:12:22.986811 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.986774 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" event={"ID":"4070bffc-3574-4ee1-b444-e405af814c82","Type":"ContainerStarted","Data":"ab77304f3a90305cc11fbe0ba43694d858a760ebe62c02b137ec514d418a6529"} Oct 30 07:12:22.987026 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:22.986840 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" event={"ID":"4070bffc-3574-4ee1-b444-e405af814c82","Type":"ContainerStarted","Data":"c1e6d08deb24261d1fe8ba5781b56e012344f5b73e91e19273e4a5eb885b0607"} Oct 30 07:12:23.028675 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:12:23.028635 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:12:23.034407 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:23.034355 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" podStartSLOduration=1.034336565 podStartE2EDuration="1.034336565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:23.030996943 +0000 UTC m=+949.374044927" watchObservedRunningTime="2026-10-30 07:12:23.034336565 +0000 UTC m=+949.377384536" Oct 30 07:12:23.500598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:23.500540 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:12:23.502681 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:23.502647 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 07:12:23.502803 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:23.502696 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" podUID="4070bffc-3574-4ee1-b444-e405af814c82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 07:12:24.501259 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:24.501223 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 07:12:24.501692 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:24.501280 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" podUID="4070bffc-3574-4ee1-b444-e405af814c82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 07:12:25.501179 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:25.501143 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 07:12:25.501330 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:25.501211 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" podUID="4070bffc-3574-4ee1-b444-e405af814c82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 07:12:26.501229 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:26.501190 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 07:12:26.501451 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:26.501261 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" podUID="4070bffc-3574-4ee1-b444-e405af814c82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 07:12:27.500856 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:27.500819 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 07:12:27.501053 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:27.500882 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" podUID="4070bffc-3574-4ee1-b444-e405af814c82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 07:12:28.500860 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:28.500824 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 07:12:28.501355 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:28.500878 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" podUID="4070bffc-3574-4ee1-b444-e405af814c82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 07:12:29.500635 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:29.500590 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 07:12:29.500810 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:29.500670 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" podUID="4070bffc-3574-4ee1-b444-e405af814c82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 07:12:30.392814 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:30.392778 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:12:30.405922 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:30.405894 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:12:30.504994 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:30.504963 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:12:30.505397 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:30.505373 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:12:30.506922 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:30.506902 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:12:31.066905 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:31.066876 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:12:43.937563 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:43.937521 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2"] Oct 30 07:12:43.943186 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:43.943156 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:12:43.959079 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:43.959056 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a0b3f5\"/\"distinct-2-w1-a0b3f5-kserve-self-signed-certs\"" Oct 30 07:12:43.972967 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:43.972940 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2"] Oct 30 07:12:44.032790 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:44.032750 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-home\") pod \"distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2\" (UID: \"e4c39d5b-24e5-433d-a50f-b7e5f45b993f\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:12:44.032978 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:44.032821 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-model-cache\") pod \"distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2\" (UID: \"e4c39d5b-24e5-433d-a50f-b7e5f45b993f\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:12:44.032978 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:44.032927 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvd7l\" (UniqueName: \"kubernetes.io/projected/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-kube-api-access-cvd7l\") pod \"distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2\" (UID: \"e4c39d5b-24e5-433d-a50f-b7e5f45b993f\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:12:44.032978 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:44.032969 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-dshm\") pod \"distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2\" (UID: \"e4c39d5b-24e5-433d-a50f-b7e5f45b993f\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:12:44.033172 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:44.033083 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-tls-certs\") pod \"distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2\" (UID: \"e4c39d5b-24e5-433d-a50f-b7e5f45b993f\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:12:44.033172 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:44.033137 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-tmp-dir\") pod \"distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2\" (UID: \"e4c39d5b-24e5-433d-a50f-b7e5f45b993f\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:12:44.134660 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:44.134624 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-tmp-dir\") pod \"distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2\" (UID: \"e4c39d5b-24e5-433d-a50f-b7e5f45b993f\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:12:44.134848 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:44.134729 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-home\") pod \"distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2\" (UID: \"e4c39d5b-24e5-433d-a50f-b7e5f45b993f\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:12:44.134848 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:44.134830 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-model-cache\") pod \"distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2\" (UID: \"e4c39d5b-24e5-433d-a50f-b7e5f45b993f\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:12:44.134948 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:44.134894 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvd7l\" (UniqueName: \"kubernetes.io/projected/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-kube-api-access-cvd7l\") pod \"distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2\" (UID: \"e4c39d5b-24e5-433d-a50f-b7e5f45b993f\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:12:44.134948 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:44.134938 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-dshm\") pod \"distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2\" (UID: \"e4c39d5b-24e5-433d-a50f-b7e5f45b993f\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:12:44.135025 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:44.135015 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-tls-certs\") pod \"distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2\" (UID: \"e4c39d5b-24e5-433d-a50f-b7e5f45b993f\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:12:44.135109 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:44.135083 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-tmp-dir\") pod \"distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2\" (UID: \"e4c39d5b-24e5-433d-a50f-b7e5f45b993f\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:12:44.135433 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:44.135408 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-model-cache\") pod \"distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2\" (UID: \"e4c39d5b-24e5-433d-a50f-b7e5f45b993f\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:12:44.136069 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:44.136043 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-home\") pod \"distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2\" (UID: \"e4c39d5b-24e5-433d-a50f-b7e5f45b993f\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:12:44.139359 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:44.139081 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-tls-certs\") pod \"distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2\" (UID: \"e4c39d5b-24e5-433d-a50f-b7e5f45b993f\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:12:44.194809 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:44.193978 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-dshm\") pod \"distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2\" (UID: \"e4c39d5b-24e5-433d-a50f-b7e5f45b993f\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:12:44.240138 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:44.239779 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvd7l\" (UniqueName: \"kubernetes.io/projected/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-kube-api-access-cvd7l\") pod \"distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2\" (UID: \"e4c39d5b-24e5-433d-a50f-b7e5f45b993f\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:12:44.259988 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:44.259953 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:12:44.600774 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:44.600692 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2"] Oct 30 07:12:45.190678 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:45.190625 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" event={"ID":"e4c39d5b-24e5-433d-a50f-b7e5f45b993f","Type":"ContainerStarted","Data":"ef6ab34e6e1b3c8dcd372d5340fd5d0f266a6fc7fe4d917cb42b96a27938bcdc"} Oct 30 07:12:45.191140 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:45.190686 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" event={"ID":"e4c39d5b-24e5-433d-a50f-b7e5f45b993f","Type":"ContainerStarted","Data":"6bd4ddb7a1e6e14cf3b6bd8315d7789db361fcb606f28314f06a1e1407cc8220"} Oct 30 07:12:45.233090 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:45.233037 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podStartSLOduration=2.233020829 podStartE2EDuration="2.233020829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:45.23035338 +0000 UTC m=+971.573401376" watchObservedRunningTime="2026-10-30 07:12:45.233020829 +0000 UTC m=+971.576068793" Oct 30 07:12:45.261265 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:45.261227 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:12:52.695886 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:52.694747 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:12:52.917104 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:52.916010 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 07:12:52.948130 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:52.947698 4951 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:12:52.948130 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:52.947771 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" podUID="de840f64-9d0c-4e5b-aa38-19c054fc8b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:12:53.056725 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:53.055753 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:12:53.190242 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:12:53.190151 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:12:54.260480 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:54.260447 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:12:54.277742 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:54.277705 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:12:54.307508 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:54.307479 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:12:59.083727 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.083688 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7"] Oct 30 07:12:59.103458 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.103426 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:12:59.123494 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.123455 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7"] Oct 30 07:12:59.129388 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.129356 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-workload-certs\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:12:59.129563 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.129405 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istio-token\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:12:59.129563 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.129450 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn52f\" (UniqueName: \"kubernetes.io/projected/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-kube-api-access-bn52f\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:12:59.129563 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.129498 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istiod-ca-cert\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:12:59.129563 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.129530 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istio-podinfo\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:12:59.129850 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.129624 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-workload-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:12:59.129850 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.129668 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istio-envoy\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:12:59.129850 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.129745 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-credential-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:12:59.129850 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.129788 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istio-data\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:12:59.230607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.230540 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istiod-ca-cert\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:12:59.230796 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.230622 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istio-podinfo\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:12:59.230796 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.230677 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-workload-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:12:59.230796 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.230695 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istio-envoy\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:12:59.230796 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.230741 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-credential-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:12:59.230796 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.230772 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istio-data\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:12:59.231054 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.230886 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-workload-certs\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:12:59.231054 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.230920 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istio-token\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:12:59.231054 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.230962 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn52f\" (UniqueName: \"kubernetes.io/projected/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-kube-api-access-bn52f\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:12:59.231208 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.231113 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-workload-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:12:59.231208 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.231164 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-credential-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:12:59.231400 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.231365 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-workload-certs\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:12:59.231550 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.231400 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istio-data\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:12:59.231550 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.231424 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istiod-ca-cert\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:12:59.233211 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.233188 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istio-envoy\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:12:59.233525 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.233504 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istio-podinfo\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:12:59.253147 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.253112 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn52f\" (UniqueName: \"kubernetes.io/projected/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-kube-api-access-bn52f\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:12:59.253289 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.253270 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istio-token\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:12:59.389641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.389510 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk"] Oct 30 07:12:59.394740 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.394714 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:12:59.409230 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.409194 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk"] Oct 30 07:12:59.419622 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.419438 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r"] Oct 30 07:12:59.424368 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.424340 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:12:59.430280 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.430255 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:12:59.451653 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.445631 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r"] Oct 30 07:12:59.451653 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.447260 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98e03afc-4edf-499e-a63a-0f92372c65a9-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:12:59.451653 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.447305 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:12:59.451653 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.447331 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:12:59.451653 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.447369 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfg4r\" (UniqueName: \"kubernetes.io/projected/98e03afc-4edf-499e-a63a-0f92372c65a9-kube-api-access-hfg4r\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:12:59.451653 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.447400 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:12:59.451653 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.447417 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98e03afc-4edf-499e-a63a-0f92372c65a9-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:12:59.451653 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.447444 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:12:59.451653 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.447459 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98e03afc-4edf-499e-a63a-0f92372c65a9-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:12:59.451653 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.447504 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:12:59.549264 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.548911 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:12:59.549264 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.548969 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:12:59.549264 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.549011 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d159ec3e-d629-4a66-b0c7-17d00455c088-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:12:59.549264 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.549077 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl4jd\" (UniqueName: \"kubernetes.io/projected/d159ec3e-d629-4a66-b0c7-17d00455c088-kube-api-access-gl4jd\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:12:59.549264 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.549106 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98e03afc-4edf-499e-a63a-0f92372c65a9-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:12:59.549264 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.549254 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:12:59.549724 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.549337 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:12:59.549724 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.549394 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d159ec3e-d629-4a66-b0c7-17d00455c088-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:12:59.549724 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.549521 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfg4r\" (UniqueName: \"kubernetes.io/projected/98e03afc-4edf-499e-a63a-0f92372c65a9-kube-api-access-hfg4r\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:12:59.549888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.549848 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:12:59.549974 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.549949 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98e03afc-4edf-499e-a63a-0f92372c65a9-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:12:59.550161 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.550135 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:12:59.550266 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.550188 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:12:59.550266 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.550223 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:12:59.550266 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.550262 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98e03afc-4edf-499e-a63a-0f92372c65a9-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:12:59.550432 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.550304 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:12:59.550432 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.550341 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:12:59.550432 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.550364 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98e03afc-4edf-499e-a63a-0f92372c65a9-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:12:59.550608 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.550430 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:12:59.550608 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.550465 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:12:59.550608 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.550505 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d159ec3e-d629-4a66-b0c7-17d00455c088-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:12:59.550608 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.550553 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:12:59.550817 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.550650 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:12:59.552900 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.552831 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:12:59.553345 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.552966 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98e03afc-4edf-499e-a63a-0f92372c65a9-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:12:59.574201 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.574165 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfg4r\" (UniqueName: \"kubernetes.io/projected/98e03afc-4edf-499e-a63a-0f92372c65a9-kube-api-access-hfg4r\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:12:59.574375 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.574213 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98e03afc-4edf-499e-a63a-0f92372c65a9-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:12:59.651622 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.651520 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:12:59.651622 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.651560 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d159ec3e-d629-4a66-b0c7-17d00455c088-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:12:59.651852 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.651654 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl4jd\" (UniqueName: \"kubernetes.io/projected/d159ec3e-d629-4a66-b0c7-17d00455c088-kube-api-access-gl4jd\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:12:59.651852 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.651755 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d159ec3e-d629-4a66-b0c7-17d00455c088-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:12:59.651852 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.651836 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:12:59.651997 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.651898 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:12:59.651997 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.651927 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:12:59.651997 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.651961 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:12:59.651997 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.651987 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:12:59.652189 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.652022 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d159ec3e-d629-4a66-b0c7-17d00455c088-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:12:59.652239 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.652177 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:12:59.652329 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.652307 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:12:59.652385 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.652369 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:12:59.652437 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.652403 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d159ec3e-d629-4a66-b0c7-17d00455c088-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:12:59.654419 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.654396 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:12:59.654540 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.654517 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d159ec3e-d629-4a66-b0c7-17d00455c088-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:12:59.674034 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.673988 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d159ec3e-d629-4a66-b0c7-17d00455c088-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:12:59.674386 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.674362 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl4jd\" (UniqueName: \"kubernetes.io/projected/d159ec3e-d629-4a66-b0c7-17d00455c088-kube-api-access-gl4jd\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:12:59.711220 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.710998 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:12:59.740738 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.740710 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:12:59.823846 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.823798 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7"] Oct 30 07:12:59.833584 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:12:59.833534 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f0d4540_46c9_4f4f_85d2_c9ee3880efa0.slice/crio-2e5d12bf5f27d084b874ed610a25d85b26d84b3ed3fed2e36a6153c791ce0f0c WatchSource:0}: Error finding container 2e5d12bf5f27d084b874ed610a25d85b26d84b3ed3fed2e36a6153c791ce0f0c: Status 404 returned error can't find the container with id 2e5d12bf5f27d084b874ed610a25d85b26d84b3ed3fed2e36a6153c791ce0f0c Oct 30 07:12:59.837294 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.837211 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:12:59.837944 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.837910 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:12:59.838049 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:12:59.837978 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:00.035738 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:00.035669 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk"] Oct 30 07:13:00.038379 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:13:00.038344 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e03afc_4edf_499e_a63a_0f92372c65a9.slice/crio-06d6d2deaf77aca3ff0e5dbe185311cdfceaa744aaa521a8ed675d6a5d3ecce4 WatchSource:0}: Error finding container 06d6d2deaf77aca3ff0e5dbe185311cdfceaa744aaa521a8ed675d6a5d3ecce4: Status 404 returned error can't find the container with id 06d6d2deaf77aca3ff0e5dbe185311cdfceaa744aaa521a8ed675d6a5d3ecce4 Oct 30 07:13:00.041907 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:00.041866 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:00.041994 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:00.041963 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:00.042031 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:00.042000 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:00.069033 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:00.069001 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r"] Oct 30 07:13:00.069848 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:13:00.069816 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd159ec3e_d629_4a66_b0c7_17d00455c088.slice/crio-5609c3aefda890e80c98e07f622113f8bc266680fe9177cb25dc15d177e1b8b3 WatchSource:0}: Error finding container 5609c3aefda890e80c98e07f622113f8bc266680fe9177cb25dc15d177e1b8b3: Status 404 returned error can't find the container with id 5609c3aefda890e80c98e07f622113f8bc266680fe9177cb25dc15d177e1b8b3 Oct 30 07:13:00.072548 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:00.072506 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:00.072676 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:00.072622 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:00.072735 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:00.072672 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:00.364084 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:00.364042 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" event={"ID":"98e03afc-4edf-499e-a63a-0f92372c65a9","Type":"ContainerStarted","Data":"7d2ee23c69b3c2cd6bfed0b70f9dbe24fbf20a11062dc61c7a36a682ce783262"} Oct 30 07:13:00.364601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:00.364111 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" event={"ID":"98e03afc-4edf-499e-a63a-0f92372c65a9","Type":"ContainerStarted","Data":"06d6d2deaf77aca3ff0e5dbe185311cdfceaa744aaa521a8ed675d6a5d3ecce4"} Oct 30 07:13:00.367002 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:00.366952 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" event={"ID":"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0","Type":"ContainerStarted","Data":"18c5c96db1706f6dd628297320cbad8a4587de62addb17a15014f99a449c2d90"} Oct 30 07:13:00.367002 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:00.366998 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" event={"ID":"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0","Type":"ContainerStarted","Data":"2e5d12bf5f27d084b874ed610a25d85b26d84b3ed3fed2e36a6153c791ce0f0c"} Oct 30 07:13:00.369418 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:00.369394 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" event={"ID":"d159ec3e-d629-4a66-b0c7-17d00455c088","Type":"ContainerStarted","Data":"048063b3f194485da85239b7e2ab0b91ec2139d12b8119e9b5f195b3884714eb"} Oct 30 07:13:00.369511 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:00.369437 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" event={"ID":"d159ec3e-d629-4a66-b0c7-17d00455c088","Type":"ContainerStarted","Data":"5609c3aefda890e80c98e07f622113f8bc266680fe9177cb25dc15d177e1b8b3"} Oct 30 07:13:00.416984 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:00.416913 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" podStartSLOduration=1.416868879 podStartE2EDuration="1.416868879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:00.41001886 +0000 UTC m=+986.753066854" watchObservedRunningTime="2026-10-30 07:13:00.416868879 +0000 UTC m=+986.759916853" Oct 30 07:13:00.464781 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:00.441001 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:13:00.465776 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:00.465683 4951 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 07:13:00.466011 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:00.465983 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" podUID="2f0d4540-46c9-4f4f-85d2-c9ee3880efa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 07:13:00.476631 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:00.469641 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" podStartSLOduration=1.469623038 podStartE2EDuration="1.469623038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:00.466959113 +0000 UTC m=+986.810007080" watchObservedRunningTime="2026-10-30 07:13:00.469623038 +0000 UTC m=+986.812671004" Oct 30 07:13:00.712266 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:00.712162 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:13:00.747594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:00.746980 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:13:01.441607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:01.438940 4951 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 07:13:01.441607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:01.439005 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" podUID="2f0d4540-46c9-4f4f-85d2-c9ee3880efa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 07:13:01.719525 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:01.718291 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 07:13:01.719525 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:01.718362 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" podUID="98e03afc-4edf-499e-a63a-0f92372c65a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 07:13:01.748598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:01.747735 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 07:13:01.748598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:01.747836 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" podUID="d159ec3e-d629-4a66-b0c7-17d00455c088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 07:13:02.443597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:02.434013 4951 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 07:13:02.443597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:02.434078 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" podUID="2f0d4540-46c9-4f4f-85d2-c9ee3880efa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 07:13:02.714076 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:02.713619 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 07:13:02.714076 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:02.713682 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" podUID="98e03afc-4edf-499e-a63a-0f92372c65a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 07:13:02.746975 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:02.746941 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 07:13:02.747256 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:02.747221 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" podUID="d159ec3e-d629-4a66-b0c7-17d00455c088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 07:13:03.466596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:03.464778 4951 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 07:13:03.466596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:03.464845 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" podUID="2f0d4540-46c9-4f4f-85d2-c9ee3880efa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 07:13:03.713232 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:03.713196 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 07:13:03.713526 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:03.713498 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" podUID="98e03afc-4edf-499e-a63a-0f92372c65a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 07:13:03.742092 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:03.741996 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 07:13:03.742358 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:03.742327 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" podUID="d159ec3e-d629-4a66-b0c7-17d00455c088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 07:13:04.463596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:04.462709 4951 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 07:13:04.463596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:04.462775 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" podUID="2f0d4540-46c9-4f4f-85d2-c9ee3880efa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 07:13:04.743656 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:04.743012 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 07:13:04.743656 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:04.743077 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" podUID="98e03afc-4edf-499e-a63a-0f92372c65a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 07:13:04.743656 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:04.743514 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 07:13:04.743656 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:04.743547 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" podUID="d159ec3e-d629-4a66-b0c7-17d00455c088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 07:13:05.442598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:05.432047 4951 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 07:13:05.442598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:05.432112 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" podUID="2f0d4540-46c9-4f4f-85d2-c9ee3880efa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 07:13:05.716492 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:05.716086 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 07:13:05.716492 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:05.716138 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" podUID="98e03afc-4edf-499e-a63a-0f92372c65a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 07:13:05.809777 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:05.809729 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 07:13:05.810409 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:05.810371 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" podUID="d159ec3e-d629-4a66-b0c7-17d00455c088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 07:13:06.430991 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:06.430953 4951 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 07:13:06.431278 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:06.431248 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" podUID="2f0d4540-46c9-4f4f-85d2-c9ee3880efa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 07:13:06.713081 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:06.712984 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 07:13:06.714680 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:06.714642 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" podUID="98e03afc-4edf-499e-a63a-0f92372c65a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 07:13:06.743036 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:06.743001 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 07:13:06.743308 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:06.743275 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" podUID="d159ec3e-d629-4a66-b0c7-17d00455c088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 07:13:07.315062 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.314994 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" podStartSLOduration=8.314973078 podStartE2EDuration="8.314973078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:00.541276512 +0000 UTC m=+986.884324478" watchObservedRunningTime="2026-10-30 07:13:07.314973078 +0000 UTC m=+993.658021043" Oct 30 07:13:07.318104 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.318065 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td"] Oct 30 07:13:07.324958 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.324923 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:13:07.364028 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.363991 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td"] Oct 30 07:13:07.428419 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.428368 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:13:07.428615 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.428435 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxjk4\" (UniqueName: \"kubernetes.io/projected/1007f3bc-edf3-41a8-b5cf-00bac3178407-kube-api-access-wxjk4\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:13:07.428615 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.428501 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:13:07.428615 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.428554 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:13:07.428838 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.428616 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:13:07.428838 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.428764 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1007f3bc-edf3-41a8-b5cf-00bac3178407-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:13:07.428838 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.428809 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:13:07.428993 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.428845 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1007f3bc-edf3-41a8-b5cf-00bac3178407-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:13:07.428993 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.428925 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1007f3bc-edf3-41a8-b5cf-00bac3178407-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:13:07.432651 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.432561 4951 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 07:13:07.432783 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.432679 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" podUID="2f0d4540-46c9-4f4f-85d2-c9ee3880efa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 07:13:07.531040 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.530546 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:13:07.531353 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.531006 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:13:07.531518 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.531499 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:13:07.531787 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.531769 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1007f3bc-edf3-41a8-b5cf-00bac3178407-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:13:07.532424 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.531993 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:13:07.532881 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.532839 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:13:07.533013 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.532992 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:13:07.533167 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.533149 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1007f3bc-edf3-41a8-b5cf-00bac3178407-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:13:07.534183 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.534062 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1007f3bc-edf3-41a8-b5cf-00bac3178407-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:13:07.534675 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.534343 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1007f3bc-edf3-41a8-b5cf-00bac3178407-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:13:07.534910 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.534892 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:13:07.535823 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.535805 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxjk4\" (UniqueName: \"kubernetes.io/projected/1007f3bc-edf3-41a8-b5cf-00bac3178407-kube-api-access-wxjk4\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:13:07.536043 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.535754 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1007f3bc-edf3-41a8-b5cf-00bac3178407-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:13:07.536043 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.535248 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:13:07.536213 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.536189 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:13:07.539618 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.539558 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:13:07.565360 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.565231 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1007f3bc-edf3-41a8-b5cf-00bac3178407-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:13:07.576079 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.573154 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxjk4\" (UniqueName: \"kubernetes.io/projected/1007f3bc-edf3-41a8-b5cf-00bac3178407-kube-api-access-wxjk4\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:13:07.703917 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:07.703853 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:13:08.434665 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:08.434630 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td"] Oct 30 07:13:08.468594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:08.465923 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:08.468594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:08.466064 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:08.468594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:08.466120 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:08.502296 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:08.496531 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:13:08.502296 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:08.500279 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:13:08.507591 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:08.506982 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:13:08.521288 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:08.521228 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:13:08.524765 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:08.522405 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:13:08.528161 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:08.528006 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:13:08.743281 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:08.743194 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:13:08.767625 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:08.749856 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:13:08.767625 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:08.758870 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:13:08.840484 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:08.840449 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" event={"ID":"1007f3bc-edf3-41a8-b5cf-00bac3178407","Type":"ContainerStarted","Data":"728ed508cb0b801bc2352dfd313ba08a587530d2a9b97b6994d6e333276b4c38"} Oct 30 07:13:08.840691 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:08.840528 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" event={"ID":"1007f3bc-edf3-41a8-b5cf-00bac3178407","Type":"ContainerStarted","Data":"16806b000361980dd0454bef65c9fa963789bccfaaf7c037456758335e4ce4f5"} Oct 30 07:13:08.896635 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:08.896554 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" podStartSLOduration=1.896534343 podStartE2EDuration="1.896534343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:08.89487096 +0000 UTC m=+995.237918973" watchObservedRunningTime="2026-10-30 07:13:08.896534343 +0000 UTC m=+995.239582309" Oct 30 07:13:09.704943 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:09.704902 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:13:09.706982 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:09.706956 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 07:13:09.707127 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:09.707002 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" podUID="1007f3bc-edf3-41a8-b5cf-00bac3178407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 07:13:10.705256 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:10.705215 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 07:13:10.705691 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:10.705272 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" podUID="1007f3bc-edf3-41a8-b5cf-00bac3178407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 07:13:11.710639 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:11.706404 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 07:13:11.710639 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:11.706480 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" podUID="1007f3bc-edf3-41a8-b5cf-00bac3178407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 07:13:12.707162 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:12.707124 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 07:13:12.707495 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:12.707446 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" podUID="1007f3bc-edf3-41a8-b5cf-00bac3178407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 07:13:13.314085 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:13.314044 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 07:13:13.500599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:13.499683 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:13.676783 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:13.676657 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:13.708708 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:13.708661 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 07:13:13.711589 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:13.711226 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 07:13:13.711814 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:13.711785 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" podUID="1007f3bc-edf3-41a8-b5cf-00bac3178407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 07:13:13.906598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:13.896670 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 07:13:14.184604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:14.167274 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 07:13:14.315226 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:14.315186 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:14.316334 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:14.316313 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:13:14.508132 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:14.508087 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:14.508768 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:14.508742 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:13:14.678760 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:14.678686 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:14.695597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:14.684373 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:13:14.709443 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:14.704748 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 07:13:14.709705 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:14.709676 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" podUID="1007f3bc-edf3-41a8-b5cf-00bac3178407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 07:13:14.756608 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:14.751922 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:14.756608 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:14.752558 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:13:15.020601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:15.018922 4951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:15.152596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:15.148698 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:15.185601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:15.162911 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:13:15.185601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:15.173852 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:15.213603 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:15.187382 4951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Oct 30 07:13:15.688907 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:15.602267 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:13:15.688907 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:15.663686 4951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded" Oct 30 07:13:15.688907 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:15.667563 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 07:13:15.688907 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:15.669396 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 07:13:15.988503 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:15.988412 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 07:13:15.988852 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:15.988818 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" podUID="1007f3bc-edf3-41a8-b5cf-00bac3178407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 07:13:15.990274 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:15.990248 4951 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:13:15.990429 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:15.990408 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" podUID="42ea0497-3b5f-4c33-a53a-bd636a4379ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:15.990866 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:15.990826 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:15.991831 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:15.991807 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 07:13:16.054732 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:16.054678 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:13:16.150737 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:16.150694 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 07:13:16.180690 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:16.180631 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 07:13:16.232927 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:16.232896 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:13:16.722309 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:16.702949 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:16.722309 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:16.709644 4951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded" Oct 30 07:13:16.722309 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:16.709693 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 07:13:16.722309 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:16.710016 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 07:13:16.722309 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:16.711790 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" podUID="1007f3bc-edf3-41a8-b5cf-00bac3178407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 07:13:16.784765 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:16.784736 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:13:16.827601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:16.827037 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:13:17.152777 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:17.152727 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:17.162737 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:17.162707 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:13:17.175015 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:17.174922 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:13:17.705252 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:17.705214 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 07:13:17.705425 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:17.705273 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" podUID="1007f3bc-edf3-41a8-b5cf-00bac3178407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 07:13:18.704642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:18.704599 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 07:13:18.705101 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:18.704660 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" podUID="1007f3bc-edf3-41a8-b5cf-00bac3178407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 07:13:19.391335 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:19.391297 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g"] Oct 30 07:13:19.396795 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:19.396769 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:13:19.415123 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:19.415091 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g"] Oct 30 07:13:19.488701 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:19.488656 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:13:19.488891 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:19.488808 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0930dc85-7926-4ea3-b498-9f88176179ca-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:13:19.488891 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:19.488840 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:13:19.489020 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:19.488875 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0930dc85-7926-4ea3-b498-9f88176179ca-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:13:19.489065 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:19.489009 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:13:19.489123 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:19.489076 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0930dc85-7926-4ea3-b498-9f88176179ca-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:13:19.489236 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:19.489215 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:13:19.489302 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:19.489273 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:13:19.489452 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:19.489376 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6rbq\" (UniqueName: \"kubernetes.io/projected/0930dc85-7926-4ea3-b498-9f88176179ca-kube-api-access-g6rbq\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:13:19.591235 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:19.591189 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:13:19.591400 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:19.591304 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0930dc85-7926-4ea3-b498-9f88176179ca-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:13:19.591400 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:19.591378 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:13:19.591470 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:19.591426 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:13:19.591530 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:19.591513 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6rbq\" (UniqueName: \"kubernetes.io/projected/0930dc85-7926-4ea3-b498-9f88176179ca-kube-api-access-g6rbq\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:13:19.591597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:19.591559 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:13:19.591675 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:19.591645 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0930dc85-7926-4ea3-b498-9f88176179ca-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:13:19.591675 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:19.591662 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:13:19.591784 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:19.591679 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:13:19.591784 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:19.591739 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0930dc85-7926-4ea3-b498-9f88176179ca-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:13:19.591901 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:19.591789 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:13:19.592322 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:19.592299 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0930dc85-7926-4ea3-b498-9f88176179ca-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:13:19.592612 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:19.592593 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:13:19.592712 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:19.592691 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:13:19.594087 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:19.594057 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0930dc85-7926-4ea3-b498-9f88176179ca-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:13:19.594268 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:19.594250 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:13:19.612966 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:19.612943 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0930dc85-7926-4ea3-b498-9f88176179ca-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:13:19.613322 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:19.613300 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6rbq\" (UniqueName: \"kubernetes.io/projected/0930dc85-7926-4ea3-b498-9f88176179ca-kube-api-access-g6rbq\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:13:19.710960 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:19.710862 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:13:20.035522 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:20.035496 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g"] Oct 30 07:13:20.037392 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:13:20.037359 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0930dc85_7926_4ea3_b498_9f88176179ca.slice/crio-aa52395fef9d5a89b565a641bcc5f9013d6cfc04296b23dbbc469e569a6afacc WatchSource:0}: Error finding container aa52395fef9d5a89b565a641bcc5f9013d6cfc04296b23dbbc469e569a6afacc: Status 404 returned error can't find the container with id aa52395fef9d5a89b565a641bcc5f9013d6cfc04296b23dbbc469e569a6afacc Oct 30 07:13:20.041308 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:20.041263 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:20.041430 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:20.041380 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:20.041499 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:20.041426 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:20.602200 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:20.602166 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:13:20.602591 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:20.602556 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:13:20.604078 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:20.604054 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:13:20.786369 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:20.786273 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" event={"ID":"0930dc85-7926-4ea3-b498-9f88176179ca","Type":"ContainerStarted","Data":"3e4fe118c55adb34f688cb8b4c77030f018f4e4847a0f2fdb38db00f4fd7bee0"} Oct 30 07:13:20.786369 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:20.786366 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" event={"ID":"0930dc85-7926-4ea3-b498-9f88176179ca","Type":"ContainerStarted","Data":"aa52395fef9d5a89b565a641bcc5f9013d6cfc04296b23dbbc469e569a6afacc"} Oct 30 07:13:20.854678 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:20.849623 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" podStartSLOduration=1.849605656 podStartE2EDuration="1.849605656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:20.849085948 +0000 UTC m=+1007.192133914" watchObservedRunningTime="2026-10-30 07:13:20.849605656 +0000 UTC m=+1007.192653625" Oct 30 07:13:21.711124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:21.711081 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:13:21.712867 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:21.712836 4951 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 07:13:21.713012 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:21.712889 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" podUID="0930dc85-7926-4ea3-b498-9f88176179ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 07:13:22.376180 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:22.376142 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns"] Oct 30 07:13:22.380399 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:22.380373 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:13:22.456876 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:22.456844 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns"] Oct 30 07:13:22.475480 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:22.475450 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:13:22.475671 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:22.475485 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:13:22.475671 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:22.475522 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pwnt\" (UniqueName: \"kubernetes.io/projected/7e1deed7-8d57-42b0-8277-96ad692ca910-kube-api-access-6pwnt\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:13:22.475671 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:22.475614 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e1deed7-8d57-42b0-8277-96ad692ca910-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:13:22.475851 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:22.475730 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e1deed7-8d57-42b0-8277-96ad692ca910-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:13:22.475851 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:22.475814 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:13:22.475851 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:22.475840 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:13:22.476010 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:22.475875 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e1deed7-8d57-42b0-8277-96ad692ca910-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:13:22.476010 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:22.475958 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:13:22.577130 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:22.577084 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:13:22.577347 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:22.577153 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:13:22.577347 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:22.577213 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pwnt\" (UniqueName: \"kubernetes.io/projected/7e1deed7-8d57-42b0-8277-96ad692ca910-kube-api-access-6pwnt\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:13:22.577347 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:22.577254 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e1deed7-8d57-42b0-8277-96ad692ca910-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:13:22.577347 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:22.577285 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e1deed7-8d57-42b0-8277-96ad692ca910-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:13:22.577607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:22.577359 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:13:22.577607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:22.577383 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:13:22.577607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:22.577419 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e1deed7-8d57-42b0-8277-96ad692ca910-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:13:22.577607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:22.577486 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:13:22.577838 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:22.577596 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:13:22.577838 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:22.577639 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:13:22.578218 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:22.578197 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e1deed7-8d57-42b0-8277-96ad692ca910-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:13:22.578452 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:22.578426 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:13:22.578675 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:22.578655 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:13:22.579887 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:22.579868 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:13:22.580356 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:22.580335 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e1deed7-8d57-42b0-8277-96ad692ca910-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:13:22.600887 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:22.600855 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e1deed7-8d57-42b0-8277-96ad692ca910-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:13:22.617743 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:22.617711 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pwnt\" (UniqueName: \"kubernetes.io/projected/7e1deed7-8d57-42b0-8277-96ad692ca910-kube-api-access-6pwnt\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:13:22.698483 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:22.698410 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:13:22.711316 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:22.711286 4951 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 07:13:22.711423 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:22.711354 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" podUID="0930dc85-7926-4ea3-b498-9f88176179ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 07:13:23.053557 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:13:23.053526 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:13:23.054915 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:23.054888 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns"] Oct 30 07:13:23.061240 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:13:23.059400 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e1deed7_8d57_42b0_8277_96ad692ca910.slice/crio-330b5cac47e3a27001495c2b5bed949a6d04b3689789915765cfff4e37cae9d2 WatchSource:0}: Error finding container 330b5cac47e3a27001495c2b5bed949a6d04b3689789915765cfff4e37cae9d2: Status 404 returned error can't find the container with id 330b5cac47e3a27001495c2b5bed949a6d04b3689789915765cfff4e37cae9d2 Oct 30 07:13:23.064142 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:23.064098 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:23.064234 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:23.064182 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:23.064234 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:23.064212 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:23.711464 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:23.711416 4951 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 07:13:23.711919 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:23.711483 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" podUID="0930dc85-7926-4ea3-b498-9f88176179ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 07:13:23.815514 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:23.815468 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" event={"ID":"7e1deed7-8d57-42b0-8277-96ad692ca910","Type":"ContainerStarted","Data":"961594a7e4af9c81a6ce86df5f263720e9bc9723f7a6f0ed61821d248038ce50"} Oct 30 07:13:23.815701 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:23.815530 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" event={"ID":"7e1deed7-8d57-42b0-8277-96ad692ca910","Type":"ContainerStarted","Data":"330b5cac47e3a27001495c2b5bed949a6d04b3689789915765cfff4e37cae9d2"} Oct 30 07:13:23.891021 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:23.890969 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" podStartSLOduration=1.8909535339999999 podStartE2EDuration="1.890953534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:23.888687706 +0000 UTC m=+1010.231735698" watchObservedRunningTime="2026-10-30 07:13:23.890953534 +0000 UTC m=+1010.234001499" Oct 30 07:13:24.698654 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:24.698611 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:13:24.701047 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:24.701024 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 07:13:24.701124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:24.701073 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" podUID="7e1deed7-8d57-42b0-8277-96ad692ca910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 07:13:24.711335 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:24.711300 4951 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 07:13:24.711484 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:24.711377 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" podUID="0930dc85-7926-4ea3-b498-9f88176179ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 07:13:25.699598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:25.699549 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 07:13:25.699838 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:25.699615 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" podUID="7e1deed7-8d57-42b0-8277-96ad692ca910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 07:13:25.711249 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:25.711214 4951 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 07:13:25.711377 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:25.711301 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" podUID="0930dc85-7926-4ea3-b498-9f88176179ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 07:13:26.699217 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:26.699178 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 07:13:26.699640 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:26.699249 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" podUID="7e1deed7-8d57-42b0-8277-96ad692ca910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 07:13:26.711282 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:26.711241 4951 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 07:13:26.711452 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:26.711343 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" podUID="0930dc85-7926-4ea3-b498-9f88176179ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 07:13:27.701829 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:27.701688 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 07:13:27.702437 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:27.702404 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" podUID="7e1deed7-8d57-42b0-8277-96ad692ca910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 07:13:28.078798 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:28.078626 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:13:28.080907 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:28.079538 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:13:28.080907 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:28.080498 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:13:28.699097 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:28.699055 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 07:13:28.699265 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:28.699116 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" podUID="7e1deed7-8d57-42b0-8277-96ad692ca910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 07:13:29.699446 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:29.699410 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 07:13:29.699852 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:29.699470 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" podUID="7e1deed7-8d57-42b0-8277-96ad692ca910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 07:13:29.833917 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:29.833880 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f"] Oct 30 07:13:29.838564 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:29.838540 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:13:29.860062 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:29.860033 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d0dace8c-openshift-default-dockercfg-lknns\"" Oct 30 07:13:29.876043 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:29.875998 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f"] Oct 30 07:13:29.995650 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:29.992510 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv5wf\" (UniqueName: \"kubernetes.io/projected/c5cb47c6-0296-4d55-8582-f7025fa4122d-kube-api-access-tv5wf\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:13:29.995650 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:29.992600 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-workload-socket\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:13:29.995650 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:29.992637 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-credential-socket\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:13:29.995650 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:29.992668 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-istio-envoy\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:13:29.995650 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:29.992727 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5cb47c6-0296-4d55-8582-f7025fa4122d-istio-podinfo\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:13:29.995650 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:29.992775 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-workload-certs\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:13:29.995650 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:29.992896 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-istio-data\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:13:29.995650 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:29.992959 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5cb47c6-0296-4d55-8582-f7025fa4122d-istio-token\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:13:29.995650 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:29.992996 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5cb47c6-0296-4d55-8582-f7025fa4122d-istiod-ca-cert\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:13:30.094442 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:30.094401 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5cb47c6-0296-4d55-8582-f7025fa4122d-istio-podinfo\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:13:30.094817 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:30.094788 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-workload-certs\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:13:30.095102 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:30.095080 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-istio-data\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:13:30.096432 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:30.096410 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5cb47c6-0296-4d55-8582-f7025fa4122d-istio-token\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:13:30.097252 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:30.097231 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5cb47c6-0296-4d55-8582-f7025fa4122d-istiod-ca-cert\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:13:30.097751 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:30.097728 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv5wf\" (UniqueName: \"kubernetes.io/projected/c5cb47c6-0296-4d55-8582-f7025fa4122d-kube-api-access-tv5wf\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:13:30.150604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:30.110309 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-workload-socket\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:13:30.150604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:30.110351 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-credential-socket\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:13:30.150604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:30.110385 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-istio-envoy\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:13:30.150604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:30.110175 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5cb47c6-0296-4d55-8582-f7025fa4122d-istio-podinfo\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:13:30.150604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:30.097169 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-workload-certs\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:13:30.150604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:30.110980 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-workload-socket\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:13:30.150604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:30.096312 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-istio-data\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:13:30.150604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:30.111222 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-credential-socket\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:13:30.150604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:30.098541 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5cb47c6-0296-4d55-8582-f7025fa4122d-istiod-ca-cert\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:13:30.150604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:30.123324 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-istio-envoy\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:13:30.150604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:30.136349 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5cb47c6-0296-4d55-8582-f7025fa4122d-istio-token\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:13:30.155598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:30.155288 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv5wf\" (UniqueName: \"kubernetes.io/projected/c5cb47c6-0296-4d55-8582-f7025fa4122d-kube-api-access-tv5wf\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:13:30.155598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:30.155559 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:13:30.544217 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:13:30.544176 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5cb47c6_0296_4d55_8582_f7025fa4122d.slice/crio-d94db9c4296ce0ab9a61f4911bbd4d2e93ce8b3f061ee3c4b92dae487cf159f4 WatchSource:0}: Error finding container d94db9c4296ce0ab9a61f4911bbd4d2e93ce8b3f061ee3c4b92dae487cf159f4: Status 404 returned error can't find the container with id d94db9c4296ce0ab9a61f4911bbd4d2e93ce8b3f061ee3c4b92dae487cf159f4 Oct 30 07:13:30.545107 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:30.545081 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f"] Oct 30 07:13:30.547619 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:30.547551 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:30.547733 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:30.547661 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:30.547733 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:30.547709 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:30.939646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:30.929480 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" event={"ID":"c5cb47c6-0296-4d55-8582-f7025fa4122d","Type":"ContainerStarted","Data":"d017f431005c40172c00b77017f518754017d3a71d0e3a548ef09c687749382b"} Oct 30 07:13:30.939646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:30.929542 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" event={"ID":"c5cb47c6-0296-4d55-8582-f7025fa4122d","Type":"ContainerStarted","Data":"d94db9c4296ce0ab9a61f4911bbd4d2e93ce8b3f061ee3c4b92dae487cf159f4"} Oct 30 07:13:30.951635 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:30.950067 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:13:30.951635 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:30.951233 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:13:30.951922 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:30.951903 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:13:30.996263 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:30.996195 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" podStartSLOduration=1.9961784420000002 podStartE2EDuration="1.996178442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:30.983426116 +0000 UTC m=+1017.326474092" watchObservedRunningTime="2026-10-30 07:13:30.996178442 +0000 UTC m=+1017.339226476" Oct 30 07:13:31.156540 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:31.156507 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:13:31.162405 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:31.162380 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:13:31.936980 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:31.936937 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:13:31.937907 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:31.937884 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:13:35.042598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:35.041845 4951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:35.053437 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:35.053392 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:35.110597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:35.063331 4951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Oct 30 07:13:35.110933 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:35.066129 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:35.192699 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:35.192655 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 07:13:35.193343 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:35.192949 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:35.486653 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:35.485661 4951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:35.690720 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:35.690672 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 07:13:35.733597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:35.720673 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:36.108765 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:36.108704 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 07:13:36.132598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:36.109879 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:13:36.156598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:36.116675 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 07:13:36.158668 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:36.158631 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:13:36.237633 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:36.201118 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:36.237633 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:36.201616 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 07:13:36.237633 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:36.202355 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:13:36.237633 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:36.202432 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:13:36.679198 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:36.679150 4951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:36.837595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:36.790703 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 07:13:36.837595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:36.808032 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:13:36.962272 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:36.962177 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:37.234068 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:36.966214 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:13:37.264888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:37.140367 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:37.266357 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:37.216708 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 07:13:37.273603 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:37.216736 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:37.291694 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:37.217892 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:37.293320 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:37.217982 4951 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:13:37.293523 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:37.293497 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" podUID="510f006a-0511-4993-8fdc-9e5ee1d6f048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:37.359599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:37.332836 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:13:37.359599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:37.332904 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" podUID="89d560f3-70af-44f7-8084-4ca608698965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:37.863697 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:37.863648 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 07:13:37.971596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:37.970730 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 07:13:38.012083 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:38.012048 4951 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:13:38.012344 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:38.012316 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" podUID="de840f64-9d0c-4e5b-aa38-19c054fc8b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:38.050040 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:38.049991 4951 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:13:38.050508 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:38.050316 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" podUID="a97844ee-24e2-4039-80d1-559b7b9f03bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:38.278605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:38.277048 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:38.278605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:38.278268 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 07:13:38.282914 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:38.282883 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 07:13:38.306306 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:38.306263 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:38.868598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:38.867718 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:38.951135 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:38.951106 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:13:38.987643 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:38.978687 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 07:13:38.989403 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:38.989375 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:13:39.092539 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:39.092507 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:13:39.192640 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:39.192544 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:13:39.227541 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:39.227513 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:13:39.386406 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:39.386374 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:13:53.024016 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:13:53.023975 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:13:56.002598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:56.001662 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:56.870597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:56.869928 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:56.956597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:56.956056 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx"] Oct 30 07:13:56.956597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:56.956327 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" podUID="a320a4da-997e-4463-bdd2-6792b153bf72" containerName="istio-proxy" containerID="cri-o://a7b632e9f2e8bf195748fe3b74654832c63b3ed7a4fd5a07353cea024302093f" gracePeriod=30 Oct 30 07:13:56.981716 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:56.981668 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:57.316913 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:57.316871 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:57.317765 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:57.317738 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 07:13:57.371588 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:57.360997 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:57.887134 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:57.887085 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 07:13:57.907650 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:57.907619 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:13:58.010595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:58.009950 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:58.010800 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:58.010702 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:13:58.236653 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:58.223952 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 07:13:58.344597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:58.322637 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 07:13:58.344597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:58.323619 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:13:58.399594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:58.384011 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:58.537063 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:58.402706 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:58.537063 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:58.515784 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:13:58.537063 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:58.515844 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:13:58.605924 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:58.605876 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc"] Oct 30 07:13:58.874656 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:58.872916 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 07:13:58.887821 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:58.887789 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-2f845b-openshift-default-dockercfg-469h2\"" Oct 30 07:13:58.906681 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:58.906637 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc"] Oct 30 07:13:58.907025 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:58.906993 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:59.019068 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.019026 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 07:13:59.019369 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.019348 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 07:13:59.019499 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.019482 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 07:13:59.019683 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.019665 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f22099c5-3928-4f09-b587-55bedc15e727-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 07:13:59.019840 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.019826 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 07:13:59.028007 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.027975 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f22099c5-3928-4f09-b587-55bedc15e727-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 07:13:59.028254 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.028235 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f22099c5-3928-4f09-b587-55bedc15e727-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 07:13:59.028423 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.028401 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vghz8\" (UniqueName: \"kubernetes.io/projected/f22099c5-3928-4f09-b587-55bedc15e727-kube-api-access-vghz8\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 07:13:59.028675 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.028653 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 07:13:59.037624 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.022810 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 07:13:59.132607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.130050 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 07:13:59.132607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.130116 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 07:13:59.132607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.130199 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f22099c5-3928-4f09-b587-55bedc15e727-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 07:13:59.132607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.130248 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 07:13:59.132607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.130286 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f22099c5-3928-4f09-b587-55bedc15e727-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 07:13:59.132607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.130356 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f22099c5-3928-4f09-b587-55bedc15e727-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 07:13:59.132607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.130420 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vghz8\" (UniqueName: \"kubernetes.io/projected/f22099c5-3928-4f09-b587-55bedc15e727-kube-api-access-vghz8\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 07:13:59.132607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.130507 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 07:13:59.132607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.130597 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 07:13:59.132607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.131203 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 07:13:59.144608 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.134277 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f22099c5-3928-4f09-b587-55bedc15e727-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 07:13:59.144608 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.134591 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 07:13:59.147456 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.147081 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 07:13:59.147456 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.147378 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 07:13:59.262772 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.262737 4951 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:13:59.263080 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.263033 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" podUID="0930dc85-7926-4ea3-b498-9f88176179ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:59.272814 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.262895 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:59.297462 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.297432 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:13:59.334625 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.319542 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 07:13:59.334625 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.333441 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 07:13:59.375610 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.335679 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vghz8\" (UniqueName: \"kubernetes.io/projected/f22099c5-3928-4f09-b587-55bedc15e727-kube-api-access-vghz8\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 07:13:59.379682 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.376762 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f22099c5-3928-4f09-b587-55bedc15e727-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 07:13:59.381411 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.381339 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f22099c5-3928-4f09-b587-55bedc15e727-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 07:13:59.805759 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.796239 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 07:13:59.808883 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:13:59.808838 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 07:14:00.214661 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:00.195697 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:14:00.280727 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:00.280676 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 07:14:00.300611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:00.300551 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:14:00.301638 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:00.301613 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 07:14:00.345308 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:00.345263 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:14:00.537199 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:00.537166 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:14:00.884661 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:00.869512 4951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:14:00.905681 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:00.900794 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 07:14:01.199462 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:01.199368 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:14:01.203619 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:01.202874 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:14:01.207588 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:01.204657 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:14:01.234643 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:01.208478 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:14:01.321313 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:01.320671 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:14:01.321313 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:01.320793 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:14:01.321313 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:01.320833 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:14:01.338759 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:01.338724 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc"] Oct 30 07:14:01.379640 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:01.373264 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:14:01.379640 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:01.374478 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:14:01.493673 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:01.493621 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" event={"ID":"f22099c5-3928-4f09-b587-55bedc15e727","Type":"ContainerStarted","Data":"60e2b323ef530ab5c13b73a60e0ca19153980f081a810343d1de89c07d09ed22"} Oct 30 07:14:02.501505 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:02.501467 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" event={"ID":"f22099c5-3928-4f09-b587-55bedc15e727","Type":"ContainerStarted","Data":"71179fdc9d969c06970e3b5bc6f4517d98daacb5570c0bdd3ce9d68a84f05de2"} Oct 30 07:14:02.563508 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:02.563442 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" podStartSLOduration=4.563424062 podStartE2EDuration="4.563424062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:14:02.560788409 +0000 UTC m=+1048.903836395" watchObservedRunningTime="2026-10-30 07:14:02.563424062 +0000 UTC m=+1048.906472028" Oct 30 07:14:03.343596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:03.343525 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 07:14:03.361595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:03.352673 4951 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 07:14:03.361595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:03.352731 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" podUID="f22099c5-3928-4f09-b587-55bedc15e727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 07:14:04.343598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:04.338717 4951 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 07:14:04.343598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:04.338792 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" podUID="f22099c5-3928-4f09-b587-55bedc15e727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 07:14:04.560872 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:04.559947 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb"] Oct 30 07:14:04.672871 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:04.672790 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:14:04.702617 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:04.701312 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb"] Oct 30 07:14:04.735606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:04.729632 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:14:04.735883 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:04.735857 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:14:04.736058 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:04.736038 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdtj5\" (UniqueName: \"kubernetes.io/projected/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-kube-api-access-zdtj5\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:14:04.736213 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:04.736195 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:14:04.736395 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:04.736376 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:14:04.736613 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:04.736594 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:14:04.736809 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:04.736794 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:14:04.736925 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:04.736909 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:14:04.737049 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:04.737034 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:14:04.862743 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:04.849414 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:14:04.862743 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:04.849512 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:14:04.862743 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:04.849620 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:14:04.862743 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:04.849709 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:14:04.862743 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:04.849740 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:14:04.862743 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:04.849788 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:14:04.862743 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:04.849834 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:14:04.862743 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:04.849887 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:14:04.862743 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:04.849953 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdtj5\" (UniqueName: \"kubernetes.io/projected/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-kube-api-access-zdtj5\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:14:04.862743 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:04.850905 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:14:04.863413 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:04.863016 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:14:04.863413 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:04.863295 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:14:04.863525 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:04.863432 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:14:04.863754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:04.863724 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:14:04.865756 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:04.865711 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:14:04.865872 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:04.865836 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:14:04.893359 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:04.893321 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:14:04.893549 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:04.893525 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdtj5\" (UniqueName: \"kubernetes.io/projected/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-kube-api-access-zdtj5\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:14:05.072020 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.071979 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:14:05.366649 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.363340 4951 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 07:14:05.366649 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.363403 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" podUID="f22099c5-3928-4f09-b587-55bedc15e727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 07:14:05.482234 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.482012 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb"] Oct 30 07:14:05.485512 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.485474 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:14:05.487447 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.487420 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:14:05.487703 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.487679 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:14:05.567523 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.567492 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:14:05.576139 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.576107 4951 generic.go:356] "Generic (PLEG): container finished" podID="a320a4da-997e-4463-bdd2-6792b153bf72" containerID="a7b632e9f2e8bf195748fe3b74654832c63b3ed7a4fd5a07353cea024302093f" exitCode=0 Oct 30 07:14:05.576297 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.576163 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" Oct 30 07:14:05.576297 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.576190 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" event={"ID":"a320a4da-997e-4463-bdd2-6792b153bf72","Type":"ContainerDied","Data":"a7b632e9f2e8bf195748fe3b74654832c63b3ed7a4fd5a07353cea024302093f"} Oct 30 07:14:05.576297 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.576256 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx" event={"ID":"a320a4da-997e-4463-bdd2-6792b153bf72","Type":"ContainerDied","Data":"5e95a5eeb2c8b576faf78a9f65ad7c2829ab6367e21e47f436dcd8262cc630c8"} Oct 30 07:14:05.576297 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.576295 4951 scope.go:122] "RemoveContainer" containerID="a7b632e9f2e8bf195748fe3b74654832c63b3ed7a4fd5a07353cea024302093f" Oct 30 07:14:05.578676 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.578644 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" event={"ID":"f0bcd71c-9f82-4f64-81f0-1c343709cfc6","Type":"ContainerStarted","Data":"c1845a9751d57bf5ffd91d5a10b90665430f7e962cd551c8edd7c1c02f788f4c"} Oct 30 07:14:05.592391 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.592354 4951 scope.go:122] "RemoveContainer" containerID="a7b632e9f2e8bf195748fe3b74654832c63b3ed7a4fd5a07353cea024302093f" Oct 30 07:14:05.593071 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:14:05.593038 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7b632e9f2e8bf195748fe3b74654832c63b3ed7a4fd5a07353cea024302093f\": container with ID starting with a7b632e9f2e8bf195748fe3b74654832c63b3ed7a4fd5a07353cea024302093f not found: ID does not exist" containerID="a7b632e9f2e8bf195748fe3b74654832c63b3ed7a4fd5a07353cea024302093f" Oct 30 07:14:05.593185 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.593084 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7b632e9f2e8bf195748fe3b74654832c63b3ed7a4fd5a07353cea024302093f"} err="failed to get container status \"a7b632e9f2e8bf195748fe3b74654832c63b3ed7a4fd5a07353cea024302093f\": rpc error: code = NotFound desc = could not find container \"a7b632e9f2e8bf195748fe3b74654832c63b3ed7a4fd5a07353cea024302093f\": container with ID starting with a7b632e9f2e8bf195748fe3b74654832c63b3ed7a4fd5a07353cea024302093f not found: ID does not exist" Oct 30 07:14:05.684559 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.682952 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a320a4da-997e-4463-bdd2-6792b153bf72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a320a4da-997e-4463-bdd2-6792b153bf72-istiod-ca-cert\") pod \"a320a4da-997e-4463-bdd2-6792b153bf72\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " Oct 30 07:14:05.684559 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.683013 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-workload-socket\") pod \"a320a4da-997e-4463-bdd2-6792b153bf72\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " Oct 30 07:14:05.684559 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.683075 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a320a4da-997e-4463-bdd2-6792b153bf72-kube-api-access-fdw25\" (UniqueName: \"kubernetes.io/projected/a320a4da-997e-4463-bdd2-6792b153bf72-kube-api-access-fdw25\") pod \"a320a4da-997e-4463-bdd2-6792b153bf72\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " Oct 30 07:14:05.684559 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.683223 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-istio-envoy\") pod \"a320a4da-997e-4463-bdd2-6792b153bf72\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " Oct 30 07:14:05.684559 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.683315 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-workload-certs\") pod \"a320a4da-997e-4463-bdd2-6792b153bf72\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " Oct 30 07:14:05.684559 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.683370 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-credential-socket\") pod \"a320a4da-997e-4463-bdd2-6792b153bf72\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " Oct 30 07:14:05.684559 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.683407 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-istio-data\") pod \"a320a4da-997e-4463-bdd2-6792b153bf72\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " Oct 30 07:14:05.684559 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.683443 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a320a4da-997e-4463-bdd2-6792b153bf72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a320a4da-997e-4463-bdd2-6792b153bf72-istio-podinfo\") pod \"a320a4da-997e-4463-bdd2-6792b153bf72\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " Oct 30 07:14:05.684559 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.683520 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a320a4da-997e-4463-bdd2-6792b153bf72-istio-token\" (UniqueName: \"kubernetes.io/projected/a320a4da-997e-4463-bdd2-6792b153bf72-istio-token\") pod \"a320a4da-997e-4463-bdd2-6792b153bf72\" (UID: \"a320a4da-997e-4463-bdd2-6792b153bf72\") " Oct 30 07:14:05.685895 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.685631 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a320a4da-997e-4463-bdd2-6792b153bf72-istiod-ca-cert" pod "a320a4da-997e-4463-bdd2-6792b153bf72" (UID: "a320a4da-997e-4463-bdd2-6792b153bf72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:14:05.686489 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.686246 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-workload-socket" pod "a320a4da-997e-4463-bdd2-6792b153bf72" (UID: "a320a4da-997e-4463-bdd2-6792b153bf72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:14:05.686611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.686501 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-workload-certs" pod "a320a4da-997e-4463-bdd2-6792b153bf72" (UID: "a320a4da-997e-4463-bdd2-6792b153bf72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:14:05.686775 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.686693 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-istio-data" pod "a320a4da-997e-4463-bdd2-6792b153bf72" (UID: "a320a4da-997e-4463-bdd2-6792b153bf72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:14:05.686775 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.686725 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-credential-socket" pod "a320a4da-997e-4463-bdd2-6792b153bf72" (UID: "a320a4da-997e-4463-bdd2-6792b153bf72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:14:05.687049 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.687019 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a320a4da-997e-4463-bdd2-6792b153bf72-istio-token" pod "a320a4da-997e-4463-bdd2-6792b153bf72" (UID: "a320a4da-997e-4463-bdd2-6792b153bf72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:14:05.689058 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.689031 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a320a4da-997e-4463-bdd2-6792b153bf72-istio-podinfo" pod "a320a4da-997e-4463-bdd2-6792b153bf72" (UID: "a320a4da-997e-4463-bdd2-6792b153bf72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:14:05.689173 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.689130 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a320a4da-997e-4463-bdd2-6792b153bf72-kube-api-access-fdw25" pod "a320a4da-997e-4463-bdd2-6792b153bf72" (UID: "a320a4da-997e-4463-bdd2-6792b153bf72"). InnerVolumeSpecName "kube-api-access-fdw25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:14:05.690141 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.690108 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-istio-envoy" pod "a320a4da-997e-4463-bdd2-6792b153bf72" (UID: "a320a4da-997e-4463-bdd2-6792b153bf72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:14:05.786604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.786443 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:14:05.786604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.786495 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:14:05.786604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.786511 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:14:05.786604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.786525 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a320a4da-997e-4463-bdd2-6792b153bf72-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:14:05.786604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.786543 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a320a4da-997e-4463-bdd2-6792b153bf72-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:14:05.786604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.786556 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a320a4da-997e-4463-bdd2-6792b153bf72-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:14:05.794101 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.786585 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:14:05.794101 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.789708 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdw25\" (UniqueName: \"kubernetes.io/projected/a320a4da-997e-4463-bdd2-6792b153bf72-kube-api-access-fdw25\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:14:05.794101 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.789733 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a320a4da-997e-4463-bdd2-6792b153bf72-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:14:05.965603 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.965489 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx"] Oct 30 07:14:05.970454 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:05.970419 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d2cqx"] Oct 30 07:14:06.331640 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:06.331185 4951 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 07:14:06.331640 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:06.331250 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" podUID="f22099c5-3928-4f09-b587-55bedc15e727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 07:14:06.593472 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:06.593305 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" event={"ID":"f0bcd71c-9f82-4f64-81f0-1c343709cfc6","Type":"ContainerStarted","Data":"b0511698b97e751f422a33a92ac77ebebd7071af46246af28f8ffc719f46b1c2"} Oct 30 07:14:06.682598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:06.672513 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" podStartSLOduration=2.672491659 podStartE2EDuration="2.672491659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:14:06.656876664 +0000 UTC m=+1052.999924631" watchObservedRunningTime="2026-10-30 07:14:06.672491659 +0000 UTC m=+1053.015539625" Oct 30 07:14:07.078001 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:07.077510 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:14:07.078001 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:07.077676 4951 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 07:14:07.078001 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:07.077722 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" podUID="f0bcd71c-9f82-4f64-81f0-1c343709cfc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 07:14:07.331557 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:07.331019 4951 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 07:14:07.331557 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:07.331084 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" podUID="f22099c5-3928-4f09-b587-55bedc15e727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 07:14:07.790734 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:07.790700 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a320a4da-997e-4463-bdd2-6792b153bf72" path="/var/lib/kubelet/pods/a320a4da-997e-4463-bdd2-6792b153bf72/volumes" Oct 30 07:14:08.078083 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:08.077989 4951 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 07:14:08.078350 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:08.078317 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" podUID="f0bcd71c-9f82-4f64-81f0-1c343709cfc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 07:14:08.337105 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:08.337003 4951 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 07:14:08.337386 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:08.337353 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" podUID="f22099c5-3928-4f09-b587-55bedc15e727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 07:14:09.073013 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:09.072958 4951 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 07:14:09.073549 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:09.073032 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" podUID="f0bcd71c-9f82-4f64-81f0-1c343709cfc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 07:14:09.330017 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:09.329931 4951 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 07:14:09.330287 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:09.330258 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" podUID="f22099c5-3928-4f09-b587-55bedc15e727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 07:14:10.072607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:10.072527 4951 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 07:14:10.072775 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:10.072632 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" podUID="f0bcd71c-9f82-4f64-81f0-1c343709cfc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 07:14:10.362955 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:10.362856 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 07:14:10.364393 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:10.364225 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 07:14:10.368595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:10.366627 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 07:14:11.073327 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:11.073243 4951 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 07:14:11.073327 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:11.073301 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" podUID="f0bcd71c-9f82-4f64-81f0-1c343709cfc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 07:14:11.663126 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:11.663091 4951 generic.go:356] "Generic (PLEG): container finished" podID="c5cb47c6-0296-4d55-8582-f7025fa4122d" containerID="d017f431005c40172c00b77017f518754017d3a71d0e3a548ef09c687749382b" exitCode=0 Oct 30 07:14:11.663676 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:11.663160 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" event={"ID":"c5cb47c6-0296-4d55-8582-f7025fa4122d","Type":"ContainerDied","Data":"d017f431005c40172c00b77017f518754017d3a71d0e3a548ef09c687749382b"} Oct 30 07:14:11.663879 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:11.663860 4951 scope.go:122] "RemoveContainer" containerID="d017f431005c40172c00b77017f518754017d3a71d0e3a548ef09c687749382b" Oct 30 07:14:11.665330 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:11.665303 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:14:11.665406 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:11.665379 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:14:11.665458 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:11.665412 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:14:12.003006 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.002970 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk"] Oct 30 07:14:12.003959 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.003932 4951 cpu_manager.go:414] "containerMap: removing container" podUID="a320a4da-997e-4463-bdd2-6792b153bf72" containerName="istio-proxy" Oct 30 07:14:12.003959 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.003959 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a320a4da-997e-4463-bdd2-6792b153bf72" containerName="istio-proxy" Oct 30 07:14:12.004125 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.004114 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="a320a4da-997e-4463-bdd2-6792b153bf72" containerName="istio-proxy" Oct 30 07:14:12.007765 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.007746 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:14:12.046635 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.045786 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk"] Oct 30 07:14:12.080227 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.080193 4951 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 07:14:12.080387 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.080250 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" podUID="f0bcd71c-9f82-4f64-81f0-1c343709cfc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 07:14:12.168261 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.168222 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-bdtxk\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:14:12.168418 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.168342 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-bdtxk\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:14:12.168418 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.168386 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-bdtxk\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:14:12.168581 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.168553 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1d68158-e979-4969-9918-e75a33ec299a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-bdtxk\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:14:12.168640 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.168617 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-bdtxk\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:14:12.168768 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.168737 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6dcd\" (UniqueName: \"kubernetes.io/projected/f1d68158-e979-4969-9918-e75a33ec299a-kube-api-access-f6dcd\") pod \"maas-default-gateway-openshift-default-6dcf757899-bdtxk\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:14:12.168832 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.168799 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-bdtxk\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:14:12.168832 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.168825 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1d68158-e979-4969-9918-e75a33ec299a-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-bdtxk\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:14:12.168923 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.168844 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1d68158-e979-4969-9918-e75a33ec299a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-bdtxk\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:14:12.270465 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.270372 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-bdtxk\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:14:12.270465 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.270443 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-bdtxk\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:14:12.270755 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.270489 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-bdtxk\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:14:12.270755 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.270652 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1d68158-e979-4969-9918-e75a33ec299a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-bdtxk\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:14:12.270755 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.270688 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-bdtxk\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:14:12.270755 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.270714 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6dcd\" (UniqueName: \"kubernetes.io/projected/f1d68158-e979-4969-9918-e75a33ec299a-kube-api-access-f6dcd\") pod \"maas-default-gateway-openshift-default-6dcf757899-bdtxk\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:14:12.270755 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.270742 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-bdtxk\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:14:12.271026 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.270775 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1d68158-e979-4969-9918-e75a33ec299a-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-bdtxk\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:14:12.271026 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.270805 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1d68158-e979-4969-9918-e75a33ec299a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-bdtxk\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:14:12.271026 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.270913 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-bdtxk\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:14:12.271306 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.271278 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-bdtxk\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:14:12.271400 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.271300 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-bdtxk\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:14:12.271534 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.271511 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-bdtxk\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:14:12.271856 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.271824 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1d68158-e979-4969-9918-e75a33ec299a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-bdtxk\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:14:12.273363 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.273340 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-bdtxk\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:14:12.274082 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.274063 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1d68158-e979-4969-9918-e75a33ec299a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-bdtxk\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:14:12.293613 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.293588 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1d68158-e979-4969-9918-e75a33ec299a-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-bdtxk\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:14:12.293706 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.293622 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6dcd\" (UniqueName: \"kubernetes.io/projected/f1d68158-e979-4969-9918-e75a33ec299a-kube-api-access-f6dcd\") pod \"maas-default-gateway-openshift-default-6dcf757899-bdtxk\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:14:12.321332 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.321304 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:14:12.672964 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.672876 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" event={"ID":"c5cb47c6-0296-4d55-8582-f7025fa4122d","Type":"ContainerStarted","Data":"efe7bf684de0aaea56de8f8d5aed614a62ff7de16ebaa88dbd4e9d8f83b7c425"} Oct 30 07:14:12.679302 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.679277 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk"] Oct 30 07:14:12.681115 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:14:12.681087 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d68158_e979_4969_9918_e75a33ec299a.slice/crio-7fe9e45aafb64b75e0213b5e7ded98ec1b5d74478ceceb4d3b83c5c989d82fbb WatchSource:0}: Error finding container 7fe9e45aafb64b75e0213b5e7ded98ec1b5d74478ceceb4d3b83c5c989d82fbb: Status 404 returned error can't find the container with id 7fe9e45aafb64b75e0213b5e7ded98ec1b5d74478ceceb4d3b83c5c989d82fbb Oct 30 07:14:12.684147 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.684116 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:14:12.684244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.684178 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:14:12.684244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:12.684215 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:14:13.074601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:13.073145 4951 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 07:14:13.074601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:13.073211 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" podUID="f0bcd71c-9f82-4f64-81f0-1c343709cfc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 07:14:13.165600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:13.164771 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:14:13.174608 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:13.171717 4951 patch_prober.go:29] interesting pod/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 07:14:13.174608 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:13.171783 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" podUID="c5cb47c6-0296-4d55-8582-f7025fa4122d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 07:14:13.705615 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:13.704557 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" event={"ID":"f1d68158-e979-4969-9918-e75a33ec299a","Type":"ContainerStarted","Data":"2e300a806dbdbbc7cc5770848bbbfc367f514e3fdce4df592cd3c21dc9e51a10"} Oct 30 07:14:13.705615 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:13.704660 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" event={"ID":"f1d68158-e979-4969-9918-e75a33ec299a","Type":"ContainerStarted","Data":"7fe9e45aafb64b75e0213b5e7ded98ec1b5d74478ceceb4d3b83c5c989d82fbb"} Oct 30 07:14:13.763596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:13.762681 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" podStartSLOduration=2.762662547 podStartE2EDuration="2.762662547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:14:13.761072047 +0000 UTC m=+1060.104120037" watchObservedRunningTime="2026-10-30 07:14:13.762662547 +0000 UTC m=+1060.105710512" Oct 30 07:14:14.121281 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:14.121220 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:14:14.122027 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:14.122003 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:14:14.138262 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:14.138231 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:14:14.157594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:14.157541 4951 patch_prober.go:29] interesting pod/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 07:14:14.157894 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:14.157847 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" podUID="c5cb47c6-0296-4d55-8582-f7025fa4122d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 07:14:14.514655 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:14.460125 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:14:14.514655 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:14.488900 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-bdtxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 07:14:14.514655 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:14.488961 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" podUID="f1d68158-e979-4969-9918-e75a33ec299a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 07:14:15.160597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:15.160176 4951 patch_prober.go:29] interesting pod/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 07:14:15.160597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:15.160273 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" podUID="c5cb47c6-0296-4d55-8582-f7025fa4122d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 07:14:15.339598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:15.338776 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-bdtxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 07:14:15.339598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:15.338835 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" podUID="f1d68158-e979-4969-9918-e75a33ec299a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 07:14:16.167597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:16.159020 4951 patch_prober.go:29] interesting pod/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 07:14:16.167597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:16.159094 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" podUID="c5cb47c6-0296-4d55-8582-f7025fa4122d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 07:14:16.324596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:16.324170 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-bdtxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 07:14:16.324596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:16.324229 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" podUID="f1d68158-e979-4969-9918-e75a33ec299a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 07:14:16.943603 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:16.942266 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:14:17.162595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:17.161958 4951 patch_prober.go:29] interesting pod/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 07:14:17.162595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:17.162018 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" podUID="c5cb47c6-0296-4d55-8582-f7025fa4122d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 07:14:17.343598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:17.340977 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-bdtxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 07:14:17.343598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:17.341056 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" podUID="f1d68158-e979-4969-9918-e75a33ec299a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 07:14:18.157268 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:18.157231 4951 patch_prober.go:29] interesting pod/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 07:14:18.157475 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:18.157297 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" podUID="c5cb47c6-0296-4d55-8582-f7025fa4122d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 07:14:18.329597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:18.325935 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-bdtxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 07:14:18.329597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:18.325994 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" podUID="f1d68158-e979-4969-9918-e75a33ec299a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 07:14:19.160802 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:19.160767 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:14:19.161894 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:19.161873 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:14:19.328631 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:19.323160 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-bdtxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 07:14:19.328631 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:19.323256 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" podUID="f1d68158-e979-4969-9918-e75a33ec299a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 07:14:20.322164 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:20.322124 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-bdtxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 07:14:20.322597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:20.322205 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" podUID="f1d68158-e979-4969-9918-e75a33ec299a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 07:14:21.979036 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:21.979006 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:14:21.979439 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:21.979328 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:14:21.980314 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:21.980292 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:14:23.023558 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:14:23.023525 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:14:28.382910 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:28.382863 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 07:14:28.930705 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:28.930656 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 07:14:29.399595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:29.398698 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:14:29.400154 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:29.399626 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:14:30.350411 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:30.350278 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 07:14:30.398941 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:30.398894 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 07:14:30.810755 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:30.810684 4951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:14:30.860593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:30.859157 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:14:53.026100 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:14:53.026053 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:14:58.562913 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:58.562874 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg"] Oct 30 07:14:58.566993 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:58.566972 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:14:58.603922 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:58.603881 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg"] Oct 30 07:14:58.634974 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:58.634914 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:14:58.634974 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:58.634967 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d493c768-747e-40e3-8e65-1c7fcb582c23-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:14:58.635244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:58.635030 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d493c768-747e-40e3-8e65-1c7fcb582c23-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:14:58.635244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:58.635087 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:14:58.635244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:58.635152 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:14:58.635408 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:58.635284 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:14:58.635408 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:58.635312 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d493c768-747e-40e3-8e65-1c7fcb582c23-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:14:58.635408 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:58.635365 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzh97\" (UniqueName: \"kubernetes.io/projected/d493c768-747e-40e3-8e65-1c7fcb582c23-kube-api-access-rzh97\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:14:58.635563 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:58.635405 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:14:58.736407 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:58.736368 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:14:58.736639 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:58.736419 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d493c768-747e-40e3-8e65-1c7fcb582c23-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:14:58.736639 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:58.736465 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzh97\" (UniqueName: \"kubernetes.io/projected/d493c768-747e-40e3-8e65-1c7fcb582c23-kube-api-access-rzh97\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:14:58.736639 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:58.736501 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:14:58.736639 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:58.736607 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:14:58.736639 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:58.736636 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d493c768-747e-40e3-8e65-1c7fcb582c23-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:14:58.736966 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:58.736691 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d493c768-747e-40e3-8e65-1c7fcb582c23-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:14:58.736966 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:58.736745 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:14:58.736966 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:58.736806 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:14:58.737257 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:58.737187 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:14:58.737257 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:58.737213 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:14:58.737257 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:58.737241 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:14:58.737533 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:58.737287 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:14:58.737680 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:58.737658 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d493c768-747e-40e3-8e65-1c7fcb582c23-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:14:58.738932 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:58.738913 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:14:58.739253 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:58.739235 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d493c768-747e-40e3-8e65-1c7fcb582c23-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:14:58.759468 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:58.759435 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d493c768-747e-40e3-8e65-1c7fcb582c23-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:14:58.759649 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:58.759534 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzh97\" (UniqueName: \"kubernetes.io/projected/d493c768-747e-40e3-8e65-1c7fcb582c23-kube-api-access-rzh97\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:14:58.884138 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:58.884054 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:14:59.197853 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:59.197815 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg"] Oct 30 07:14:59.198861 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:14:59.198827 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd493c768_747e_40e3_8e65_1c7fcb582c23.slice/crio-58de8276b0c5b9d43240017721745c87d53cc261c1034d874316ae5a60a03346 WatchSource:0}: Error finding container 58de8276b0c5b9d43240017721745c87d53cc261c1034d874316ae5a60a03346: Status 404 returned error can't find the container with id 58de8276b0c5b9d43240017721745c87d53cc261c1034d874316ae5a60a03346 Oct 30 07:14:59.201476 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:59.201440 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:14:59.201567 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:59.201511 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:14:59.201567 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:59.201540 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:14:59.475714 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:59.475606 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" event={"ID":"d493c768-747e-40e3-8e65-1c7fcb582c23","Type":"ContainerStarted","Data":"a9e7cb0222c9aaa92cd97fd7012cb17b60cb8333fceca0d6d6919b45fec8d32e"} Oct 30 07:14:59.475714 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:59.475679 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" event={"ID":"d493c768-747e-40e3-8e65-1c7fcb582c23","Type":"ContainerStarted","Data":"58de8276b0c5b9d43240017721745c87d53cc261c1034d874316ae5a60a03346"} Oct 30 07:14:59.520130 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:59.520077 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" podStartSLOduration=1.520062923 podStartE2EDuration="1.520062923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:14:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:14:59.517096512 +0000 UTC m=+1105.860144480" watchObservedRunningTime="2026-10-30 07:14:59.520062923 +0000 UTC m=+1105.863110888" Oct 30 07:14:59.884833 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:14:59.884801 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:15:00.176025 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:00.175948 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq"] Oct 30 07:15:00.180186 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:00.180160 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq" Oct 30 07:15:00.208246 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:00.208209 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq"] Oct 30 07:15:00.218792 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:00.218766 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq"] Oct 30 07:15:00.219007 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:00.218842 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq" Oct 30 07:15:00.268639 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:00.268600 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq\" (UID: \"06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq" Oct 30 07:15:00.268837 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:00.268652 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfd1f8d4-cc32-47f5-9f87-954c79e326e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq\" (UID: \"bfd1f8d4-cc32-47f5-9f87-954c79e326e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq" Oct 30 07:15:00.268915 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:00.268874 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgbqf\" (UniqueName: \"kubernetes.io/projected/bfd1f8d4-cc32-47f5-9f87-954c79e326e3-kube-api-access-vgbqf\") pod \"maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq\" (UID: \"bfd1f8d4-cc32-47f5-9f87-954c79e326e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq" Oct 30 07:15:00.269035 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:00.268938 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmwsg\" (UniqueName: \"kubernetes.io/projected/06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1-kube-api-access-pmwsg\") pod \"maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq\" (UID: \"06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq" Oct 30 07:15:00.273347 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:00.273317 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq"] Oct 30 07:15:00.294297 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:00.294272 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:15:00.370287 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:00.370253 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgbqf\" (UniqueName: \"kubernetes.io/projected/bfd1f8d4-cc32-47f5-9f87-954c79e326e3-kube-api-access-vgbqf\") pod \"maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq\" (UID: \"bfd1f8d4-cc32-47f5-9f87-954c79e326e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq" Oct 30 07:15:00.370287 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:00.370292 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmwsg\" (UniqueName: \"kubernetes.io/projected/06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1-kube-api-access-pmwsg\") pod \"maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq\" (UID: \"06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq" Oct 30 07:15:00.370561 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:00.370543 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq\" (UID: \"06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq" Oct 30 07:15:00.370695 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:00.370567 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfd1f8d4-cc32-47f5-9f87-954c79e326e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq\" (UID: \"bfd1f8d4-cc32-47f5-9f87-954c79e326e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq" Oct 30 07:15:00.371226 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:00.371207 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfd1f8d4-cc32-47f5-9f87-954c79e326e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq\" (UID: \"bfd1f8d4-cc32-47f5-9f87-954c79e326e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq" Oct 30 07:15:00.371285 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:00.371249 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq\" (UID: \"06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq" Oct 30 07:15:00.392654 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:00.392626 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgbqf\" (UniqueName: \"kubernetes.io/projected/bfd1f8d4-cc32-47f5-9f87-954c79e326e3-kube-api-access-vgbqf\") pod \"maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq\" (UID: \"bfd1f8d4-cc32-47f5-9f87-954c79e326e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq" Oct 30 07:15:00.392798 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:00.392675 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmwsg\" (UniqueName: \"kubernetes.io/projected/06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1-kube-api-access-pmwsg\") pod \"maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq\" (UID: \"06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq" Oct 30 07:15:00.498627 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:00.498532 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq" Oct 30 07:15:00.600366 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:00.597413 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq" Oct 30 07:15:00.820686 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:00.820645 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq"] Oct 30 07:15:00.859079 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:15:00.859037 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06a9e4f0_da4c_4abc_a5d1_fcfa4a6bb2a1.slice/crio-cb08e619c7e0dd735292a89aa448b79383f7e53414088ec8a77231ce33d01320 WatchSource:0}: Error finding container cb08e619c7e0dd735292a89aa448b79383f7e53414088ec8a77231ce33d01320: Status 404 returned error can't find the container with id cb08e619c7e0dd735292a89aa448b79383f7e53414088ec8a77231ce33d01320 Oct 30 07:15:00.862990 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:00.862962 4951 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 07:15:00.886235 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:00.886200 4951 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 07:15:00.886696 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:00.886250 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" podUID="d493c768-747e-40e3-8e65-1c7fcb582c23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 07:15:00.956228 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:00.956199 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq"] Oct 30 07:15:00.959544 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:15:00.959508 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd1f8d4_cc32_47f5_9f87_954c79e326e3.slice/crio-35cef0110dd51b1ae592155a7d8601fa699fa599bad978d3ccaef4088ca0a3f0 WatchSource:0}: Error finding container 35cef0110dd51b1ae592155a7d8601fa699fa599bad978d3ccaef4088ca0a3f0: Status 404 returned error can't find the container with id 35cef0110dd51b1ae592155a7d8601fa699fa599bad978d3ccaef4088ca0a3f0 Oct 30 07:15:01.213689 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:01.188178 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7605-f11303"] Oct 30 07:15:01.223067 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:01.223031 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7605-f11303"] Oct 30 07:15:01.223444 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:01.223418 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7605-f11303" Oct 30 07:15:01.301702 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:01.300931 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5z8jp\"" Oct 30 07:15:01.389853 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:01.389813 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45hl4\" (UniqueName: \"kubernetes.io/projected/722bff8e-d681-405b-b0ea-dd10ae461d5e-kube-api-access-45hl4\") pod \"test-curl-7605-f11303\" (UID: \"722bff8e-d681-405b-b0ea-dd10ae461d5e\") " pod="opendatahub/test-curl-7605-f11303" Oct 30 07:15:01.489931 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:01.489828 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq" event={"ID":"06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1","Type":"ContainerStarted","Data":"cb08e619c7e0dd735292a89aa448b79383f7e53414088ec8a77231ce33d01320"} Oct 30 07:15:01.490813 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:01.490787 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45hl4\" (UniqueName: \"kubernetes.io/projected/722bff8e-d681-405b-b0ea-dd10ae461d5e-kube-api-access-45hl4\") pod \"test-curl-7605-f11303\" (UID: \"722bff8e-d681-405b-b0ea-dd10ae461d5e\") " pod="opendatahub/test-curl-7605-f11303" Oct 30 07:15:01.491308 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:01.491282 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq" event={"ID":"bfd1f8d4-cc32-47f5-9f87-954c79e326e3","Type":"ContainerStarted","Data":"35cef0110dd51b1ae592155a7d8601fa699fa599bad978d3ccaef4088ca0a3f0"} Oct 30 07:15:01.513000 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:01.512966 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45hl4\" (UniqueName: \"kubernetes.io/projected/722bff8e-d681-405b-b0ea-dd10ae461d5e-kube-api-access-45hl4\") pod \"test-curl-7605-f11303\" (UID: \"722bff8e-d681-405b-b0ea-dd10ae461d5e\") " pod="opendatahub/test-curl-7605-f11303" Oct 30 07:15:01.609153 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:01.609120 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7605-f11303" Oct 30 07:15:01.884395 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:01.884353 4951 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 07:15:01.884627 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:01.884405 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" podUID="d493c768-747e-40e3-8e65-1c7fcb582c23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 07:15:01.962490 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:01.962453 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7605-f11303"] Oct 30 07:15:02.503426 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:02.503383 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7605-f11303" event={"ID":"722bff8e-d681-405b-b0ea-dd10ae461d5e","Type":"ContainerStarted","Data":"77801decc858a19df595c26bb7a1e9c3616014b7db56652985710211999823e9"} Oct 30 07:15:02.885580 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:02.885473 4951 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 07:15:02.885580 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:02.885529 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" podUID="d493c768-747e-40e3-8e65-1c7fcb582c23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 07:15:03.884942 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:03.884879 4951 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 07:15:03.885373 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:03.884949 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" podUID="d493c768-747e-40e3-8e65-1c7fcb582c23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 07:15:04.885934 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:04.885423 4951 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 07:15:04.885934 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:04.885489 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" podUID="d493c768-747e-40e3-8e65-1c7fcb582c23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 07:15:05.886543 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:05.886440 4951 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 07:15:05.887044 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:05.886534 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" podUID="d493c768-747e-40e3-8e65-1c7fcb582c23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 07:15:06.577987 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:06.577943 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq" event={"ID":"bfd1f8d4-cc32-47f5-9f87-954c79e326e3","Type":"ContainerStarted","Data":"a6de2d61f4dbe5d2c21d17bcc89ed10a0e94a025b299c481bbf79a922d7a34d4"} Oct 30 07:15:06.579941 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:06.579899 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq" event={"ID":"06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1","Type":"ContainerStarted","Data":"4fcc00ab38082e2d9ff8d1952d2a4726f85fe0ead612c88b7b2742c41f10a92e"} Oct 30 07:15:06.581985 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:06.581948 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7605-f11303" event={"ID":"722bff8e-d681-405b-b0ea-dd10ae461d5e","Type":"ContainerStarted","Data":"4729bc486984105145ce4213e29ddb5028a6727730f921c3b08b7d151365d49f"} Oct 30 07:15:06.618734 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:06.618683 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq" podStartSLOduration=2.022102379 podStartE2EDuration="6.618669674s" totalImagesPullingTime="4.596567295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:06.61505066 +0000 UTC m=+1112.958098625" watchObservedRunningTime="2026-10-30 07:15:06.618669674 +0000 UTC m=+1112.961717638" Oct 30 07:15:06.651085 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:06.651031 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7605-f11303" podStartSLOduration=1.4506210689999999 podStartE2EDuration="5.65101207s" totalImagesPullingTime="4.200391001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:06.650600709 +0000 UTC m=+1112.993648700" watchObservedRunningTime="2026-10-30 07:15:06.65101207 +0000 UTC m=+1112.994060036" Oct 30 07:15:06.693272 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:06.693208 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq" podStartSLOduration=1.879061919 podStartE2EDuration="6.693186721s" totalImagesPullingTime="4.814124802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:06.688889407 +0000 UTC m=+1113.031937389" watchObservedRunningTime="2026-10-30 07:15:06.693186721 +0000 UTC m=+1113.036234688" Oct 30 07:15:06.889135 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:06.889049 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:15:06.889535 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:06.889320 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:15:06.890055 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:06.890038 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:15:07.567170 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:07.567135 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7605-f11303"] Oct 30 07:15:07.573834 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:07.573801 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7605-f11303"] Oct 30 07:15:07.591886 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:07.591855 4951 generic.go:356] "Generic (PLEG): container finished" podID="bfd1f8d4-cc32-47f5-9f87-954c79e326e3" containerID="a6de2d61f4dbe5d2c21d17bcc89ed10a0e94a025b299c481bbf79a922d7a34d4" exitCode=60 Oct 30 07:15:07.592139 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:07.591947 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq" event={"ID":"bfd1f8d4-cc32-47f5-9f87-954c79e326e3","Type":"ContainerDied","Data":"a6de2d61f4dbe5d2c21d17bcc89ed10a0e94a025b299c481bbf79a922d7a34d4"} Oct 30 07:15:07.592391 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:07.592364 4951 scope.go:122] "RemoveContainer" containerID="a6de2d61f4dbe5d2c21d17bcc89ed10a0e94a025b299c481bbf79a922d7a34d4" Oct 30 07:15:07.594330 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:07.594304 4951 generic.go:356] "Generic (PLEG): container finished" podID="06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1" containerID="4fcc00ab38082e2d9ff8d1952d2a4726f85fe0ead612c88b7b2742c41f10a92e" exitCode=60 Oct 30 07:15:07.594524 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:07.594497 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq" event={"ID":"06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1","Type":"ContainerDied","Data":"4fcc00ab38082e2d9ff8d1952d2a4726f85fe0ead612c88b7b2742c41f10a92e"} Oct 30 07:15:07.595066 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:07.595043 4951 scope.go:122] "RemoveContainer" containerID="4fcc00ab38082e2d9ff8d1952d2a4726f85fe0ead612c88b7b2742c41f10a92e" Oct 30 07:15:08.604036 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:08.604003 4951 generic.go:356] "Generic (PLEG): container finished" podID="06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1" containerID="141be094bb75870103f1860ab11ef397af0f01eaf09821f4ceac557476eb060c" exitCode=60 Oct 30 07:15:08.605410 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:08.604088 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq" event={"ID":"06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1","Type":"ContainerDied","Data":"141be094bb75870103f1860ab11ef397af0f01eaf09821f4ceac557476eb060c"} Oct 30 07:15:08.605410 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:08.604162 4951 scope.go:122] "RemoveContainer" containerID="4fcc00ab38082e2d9ff8d1952d2a4726f85fe0ead612c88b7b2742c41f10a92e" Oct 30 07:15:08.605410 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:08.604479 4951 scope.go:122] "RemoveContainer" containerID="141be094bb75870103f1860ab11ef397af0f01eaf09821f4ceac557476eb060c" Oct 30 07:15:08.605410 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:15:08.604912 4951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq_odh-ai-gateway-infra(06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq" podUID="06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1" Oct 30 07:15:08.606903 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:08.606875 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq" event={"ID":"bfd1f8d4-cc32-47f5-9f87-954c79e326e3","Type":"ContainerStarted","Data":"77efbacf706942ea3720dd23cc33586e01f5e3e20a0fe69bc0313919601a68a0"} Oct 30 07:15:08.607012 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:08.606958 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7605-f11303" podUID="722bff8e-d681-405b-b0ea-dd10ae461d5e" containerName="test-curl-7605-f11303" containerID="cri-o://4729bc486984105145ce4213e29ddb5028a6727730f921c3b08b7d151365d49f" gracePeriod=2 Oct 30 07:15:09.615440 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:09.615409 4951 scope.go:122] "RemoveContainer" containerID="141be094bb75870103f1860ab11ef397af0f01eaf09821f4ceac557476eb060c" Oct 30 07:15:09.615916 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:15:09.615819 4951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq_odh-ai-gateway-infra(06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq" podUID="06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1" Oct 30 07:15:09.617263 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:09.617236 4951 generic.go:356] "Generic (PLEG): container finished" podID="bfd1f8d4-cc32-47f5-9f87-954c79e326e3" containerID="77efbacf706942ea3720dd23cc33586e01f5e3e20a0fe69bc0313919601a68a0" exitCode=60 Oct 30 07:15:09.617396 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:09.617324 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq" event={"ID":"bfd1f8d4-cc32-47f5-9f87-954c79e326e3","Type":"ContainerDied","Data":"77efbacf706942ea3720dd23cc33586e01f5e3e20a0fe69bc0313919601a68a0"} Oct 30 07:15:09.617435 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:09.617396 4951 scope.go:122] "RemoveContainer" containerID="a6de2d61f4dbe5d2c21d17bcc89ed10a0e94a025b299c481bbf79a922d7a34d4" Oct 30 07:15:09.617553 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:09.617532 4951 scope.go:122] "RemoveContainer" containerID="77efbacf706942ea3720dd23cc33586e01f5e3e20a0fe69bc0313919601a68a0" Oct 30 07:15:09.617884 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:15:09.617859 4951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq_odh-ai-gateway-infra(bfd1f8d4-cc32-47f5-9f87-954c79e326e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq" podUID="bfd1f8d4-cc32-47f5-9f87-954c79e326e3" Oct 30 07:15:10.626862 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:10.626837 4951 scope.go:122] "RemoveContainer" containerID="77efbacf706942ea3720dd23cc33586e01f5e3e20a0fe69bc0313919601a68a0" Oct 30 07:15:10.627269 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:15:10.627244 4951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq_odh-ai-gateway-infra(bfd1f8d4-cc32-47f5-9f87-954c79e326e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq" podUID="bfd1f8d4-cc32-47f5-9f87-954c79e326e3" Oct 30 07:15:10.841197 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:10.841170 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7605-f11303" Oct 30 07:15:10.992667 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:10.992581 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/722bff8e-d681-405b-b0ea-dd10ae461d5e-kube-api-access-45hl4\" (UniqueName: \"kubernetes.io/projected/722bff8e-d681-405b-b0ea-dd10ae461d5e-kube-api-access-45hl4\") pod \"722bff8e-d681-405b-b0ea-dd10ae461d5e\" (UID: \"722bff8e-d681-405b-b0ea-dd10ae461d5e\") " Oct 30 07:15:10.994715 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:10.994683 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/722bff8e-d681-405b-b0ea-dd10ae461d5e-kube-api-access-45hl4" pod "722bff8e-d681-405b-b0ea-dd10ae461d5e" (UID: "722bff8e-d681-405b-b0ea-dd10ae461d5e"). InnerVolumeSpecName "kube-api-access-45hl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:15:11.094393 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:11.094349 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45hl4\" (UniqueName: \"kubernetes.io/projected/722bff8e-d681-405b-b0ea-dd10ae461d5e-kube-api-access-45hl4\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:15:11.634723 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:11.634683 4951 generic.go:356] "Generic (PLEG): container finished" podID="722bff8e-d681-405b-b0ea-dd10ae461d5e" containerID="4729bc486984105145ce4213e29ddb5028a6727730f921c3b08b7d151365d49f" exitCode=137 Oct 30 07:15:11.635215 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:11.634736 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7605-f11303" Oct 30 07:15:11.635215 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:11.634790 4951 scope.go:122] "RemoveContainer" containerID="4729bc486984105145ce4213e29ddb5028a6727730f921c3b08b7d151365d49f" Oct 30 07:15:11.644342 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:11.644322 4951 scope.go:122] "RemoveContainer" containerID="4729bc486984105145ce4213e29ddb5028a6727730f921c3b08b7d151365d49f" Oct 30 07:15:11.644697 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:15:11.644676 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4729bc486984105145ce4213e29ddb5028a6727730f921c3b08b7d151365d49f\": container with ID starting with 4729bc486984105145ce4213e29ddb5028a6727730f921c3b08b7d151365d49f not found: ID does not exist" containerID="4729bc486984105145ce4213e29ddb5028a6727730f921c3b08b7d151365d49f" Oct 30 07:15:11.644777 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:11.644708 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4729bc486984105145ce4213e29ddb5028a6727730f921c3b08b7d151365d49f"} err="failed to get container status \"4729bc486984105145ce4213e29ddb5028a6727730f921c3b08b7d151365d49f\": rpc error: code = NotFound desc = could not find container \"4729bc486984105145ce4213e29ddb5028a6727730f921c3b08b7d151365d49f\": container with ID starting with 4729bc486984105145ce4213e29ddb5028a6727730f921c3b08b7d151365d49f not found: ID does not exist" Oct 30 07:15:11.655996 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:15:11.655962 4951 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7605-f11303\" is forbidden: User \"system:node:ip-10-0-132-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-173.ec2.internal' and this object" podUID="722bff8e-d681-405b-b0ea-dd10ae461d5e" pod="opendatahub/test-curl-7605-f11303" Oct 30 07:15:11.773221 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:11.773187 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="722bff8e-d681-405b-b0ea-dd10ae461d5e" path="/var/lib/kubelet/pods/722bff8e-d681-405b-b0ea-dd10ae461d5e/volumes" Oct 30 07:15:12.068176 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:12.068141 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27d5b50e-dnxq8"] Oct 30 07:15:12.068924 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:12.068902 4951 cpu_manager.go:414] "containerMap: removing container" podUID="722bff8e-d681-405b-b0ea-dd10ae461d5e" containerName="test-curl-7605-f11303" Oct 30 07:15:12.068924 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:12.068923 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="722bff8e-d681-405b-b0ea-dd10ae461d5e" containerName="test-curl-7605-f11303" Oct 30 07:15:12.069148 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:12.069133 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="722bff8e-d681-405b-b0ea-dd10ae461d5e" containerName="test-curl-7605-f11303" Oct 30 07:15:12.074093 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:12.074071 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27d5b50e-dnxq8" Oct 30 07:15:12.092784 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:12.091613 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:15:12.106635 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:12.105346 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27d5b50e-dnxq8"] Oct 30 07:15:12.209274 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:12.209232 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv87b\" (UniqueName: \"kubernetes.io/projected/fc5c70db-419b-439d-aa07-a9bdfc78aad0-kube-api-access-wv87b\") pod \"maas-api-revoke-sub-27d5b50e-dnxq8\" (UID: \"fc5c70db-419b-439d-aa07-a9bdfc78aad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27d5b50e-dnxq8" Oct 30 07:15:12.209489 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:12.209284 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc5c70db-419b-439d-aa07-a9bdfc78aad0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27d5b50e-dnxq8\" (UID: \"fc5c70db-419b-439d-aa07-a9bdfc78aad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27d5b50e-dnxq8" Oct 30 07:15:12.209556 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:12.209506 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc5c70db-419b-439d-aa07-a9bdfc78aad0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27d5b50e-dnxq8\" (UID: \"fc5c70db-419b-439d-aa07-a9bdfc78aad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27d5b50e-dnxq8" Oct 30 07:15:12.311290 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:12.311232 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv87b\" (UniqueName: \"kubernetes.io/projected/fc5c70db-419b-439d-aa07-a9bdfc78aad0-kube-api-access-wv87b\") pod \"maas-api-revoke-sub-27d5b50e-dnxq8\" (UID: \"fc5c70db-419b-439d-aa07-a9bdfc78aad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27d5b50e-dnxq8" Oct 30 07:15:12.311290 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:12.311289 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc5c70db-419b-439d-aa07-a9bdfc78aad0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27d5b50e-dnxq8\" (UID: \"fc5c70db-419b-439d-aa07-a9bdfc78aad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27d5b50e-dnxq8" Oct 30 07:15:12.311538 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:12.311408 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc5c70db-419b-439d-aa07-a9bdfc78aad0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27d5b50e-dnxq8\" (UID: \"fc5c70db-419b-439d-aa07-a9bdfc78aad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27d5b50e-dnxq8" Oct 30 07:15:12.312461 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:12.312435 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc5c70db-419b-439d-aa07-a9bdfc78aad0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27d5b50e-dnxq8\" (UID: \"fc5c70db-419b-439d-aa07-a9bdfc78aad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27d5b50e-dnxq8" Oct 30 07:15:12.317078 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:12.317052 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc5c70db-419b-439d-aa07-a9bdfc78aad0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27d5b50e-dnxq8\" (UID: \"fc5c70db-419b-439d-aa07-a9bdfc78aad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27d5b50e-dnxq8" Oct 30 07:15:12.335116 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:12.335055 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv87b\" (UniqueName: \"kubernetes.io/projected/fc5c70db-419b-439d-aa07-a9bdfc78aad0-kube-api-access-wv87b\") pod \"maas-api-revoke-sub-27d5b50e-dnxq8\" (UID: \"fc5c70db-419b-439d-aa07-a9bdfc78aad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27d5b50e-dnxq8" Oct 30 07:15:12.402687 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:12.402649 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27d5b50e-dnxq8" Oct 30 07:15:12.716205 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:12.716170 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27d5b50e-dnxq8"] Oct 30 07:15:12.717472 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:15:12.717440 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5c70db_419b_439d_aa07_a9bdfc78aad0.slice/crio-e60939b0f36fbc2eea47534c06e885eef9c23d352153b0121215a7e9c939a050 WatchSource:0}: Error finding container e60939b0f36fbc2eea47534c06e885eef9c23d352153b0121215a7e9c939a050: Status 404 returned error can't find the container with id e60939b0f36fbc2eea47534c06e885eef9c23d352153b0121215a7e9c939a050 Oct 30 07:15:13.651314 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:13.651281 4951 generic.go:356] "Generic (PLEG): container finished" podID="fc5c70db-419b-439d-aa07-a9bdfc78aad0" containerID="2e5d0e999ed61144f7d174347904f6336a028f6c633f867369664e0fea660c8c" exitCode=0 Oct 30 07:15:13.651459 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:13.651362 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27d5b50e-dnxq8" event={"ID":"fc5c70db-419b-439d-aa07-a9bdfc78aad0","Type":"ContainerDied","Data":"2e5d0e999ed61144f7d174347904f6336a028f6c633f867369664e0fea660c8c"} Oct 30 07:15:13.651459 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:13.651413 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27d5b50e-dnxq8" event={"ID":"fc5c70db-419b-439d-aa07-a9bdfc78aad0","Type":"ContainerStarted","Data":"e60939b0f36fbc2eea47534c06e885eef9c23d352153b0121215a7e9c939a050"} Oct 30 07:15:14.825141 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:14.825099 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27d5b50e-dnxq8" Oct 30 07:15:14.962068 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:14.961976 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc5c70db-419b-439d-aa07-a9bdfc78aad0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc5c70db-419b-439d-aa07-a9bdfc78aad0-maas-api-openshift-ca\") pod \"fc5c70db-419b-439d-aa07-a9bdfc78aad0\" (UID: \"fc5c70db-419b-439d-aa07-a9bdfc78aad0\") " Oct 30 07:15:14.962068 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:14.962036 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc5c70db-419b-439d-aa07-a9bdfc78aad0-kube-api-access-wv87b\" (UniqueName: \"kubernetes.io/projected/fc5c70db-419b-439d-aa07-a9bdfc78aad0-kube-api-access-wv87b\") pod \"fc5c70db-419b-439d-aa07-a9bdfc78aad0\" (UID: \"fc5c70db-419b-439d-aa07-a9bdfc78aad0\") " Oct 30 07:15:14.962287 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:14.962088 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc5c70db-419b-439d-aa07-a9bdfc78aad0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc5c70db-419b-439d-aa07-a9bdfc78aad0-maas-api-xks-ca\") pod \"fc5c70db-419b-439d-aa07-a9bdfc78aad0\" (UID: \"fc5c70db-419b-439d-aa07-a9bdfc78aad0\") " Oct 30 07:15:14.962372 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:14.962348 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc5c70db-419b-439d-aa07-a9bdfc78aad0-maas-api-openshift-ca" pod "fc5c70db-419b-439d-aa07-a9bdfc78aad0" (UID: "fc5c70db-419b-439d-aa07-a9bdfc78aad0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:15:14.962661 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:14.962643 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc5c70db-419b-439d-aa07-a9bdfc78aad0-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:15:14.964276 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:14.964248 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc5c70db-419b-439d-aa07-a9bdfc78aad0-kube-api-access-wv87b" pod "fc5c70db-419b-439d-aa07-a9bdfc78aad0" (UID: "fc5c70db-419b-439d-aa07-a9bdfc78aad0"). InnerVolumeSpecName "kube-api-access-wv87b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:15:14.964742 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:14.964722 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc5c70db-419b-439d-aa07-a9bdfc78aad0-maas-api-xks-ca" pod "fc5c70db-419b-439d-aa07-a9bdfc78aad0" (UID: "fc5c70db-419b-439d-aa07-a9bdfc78aad0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:15:15.094597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:15.068192 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv87b\" (UniqueName: \"kubernetes.io/projected/fc5c70db-419b-439d-aa07-a9bdfc78aad0-kube-api-access-wv87b\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:15:15.095076 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:15.095053 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc5c70db-419b-439d-aa07-a9bdfc78aad0-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:15:15.668139 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:15.668057 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27d5b50e-dnxq8" Oct 30 07:15:15.668139 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:15.668083 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27d5b50e-dnxq8" event={"ID":"fc5c70db-419b-439d-aa07-a9bdfc78aad0","Type":"ContainerDied","Data":"e60939b0f36fbc2eea47534c06e885eef9c23d352153b0121215a7e9c939a050"} Oct 30 07:15:15.668139 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:15.668134 4951 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e60939b0f36fbc2eea47534c06e885eef9c23d352153b0121215a7e9c939a050" Oct 30 07:15:20.765608 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:20.764121 4951 scope.go:122] "RemoveContainer" containerID="77efbacf706942ea3720dd23cc33586e01f5e3e20a0fe69bc0313919601a68a0" Oct 30 07:15:21.756669 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:21.756621 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq" event={"ID":"bfd1f8d4-cc32-47f5-9f87-954c79e326e3","Type":"ContainerStarted","Data":"05da206ee2f8305babc4cdf8a1e89e5ef0a38add4a00a0c5512114175d71dbe9"} Oct 30 07:15:21.757174 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:21.757146 4951 scope.go:122] "RemoveContainer" containerID="05da206ee2f8305babc4cdf8a1e89e5ef0a38add4a00a0c5512114175d71dbe9" Oct 30 07:15:21.757918 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:15:21.757485 4951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq_odh-ai-gateway-infra(bfd1f8d4-cc32-47f5-9f87-954c79e326e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq" podUID="bfd1f8d4-cc32-47f5-9f87-954c79e326e3" Oct 30 07:15:22.764080 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:22.764050 4951 scope.go:122] "RemoveContainer" containerID="141be094bb75870103f1860ab11ef397af0f01eaf09821f4ceac557476eb060c" Oct 30 07:15:22.766237 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:22.766209 4951 generic.go:356] "Generic (PLEG): container finished" podID="bfd1f8d4-cc32-47f5-9f87-954c79e326e3" containerID="05da206ee2f8305babc4cdf8a1e89e5ef0a38add4a00a0c5512114175d71dbe9" exitCode=60 Oct 30 07:15:22.766349 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:22.766283 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq" event={"ID":"bfd1f8d4-cc32-47f5-9f87-954c79e326e3","Type":"ContainerDied","Data":"05da206ee2f8305babc4cdf8a1e89e5ef0a38add4a00a0c5512114175d71dbe9"} Oct 30 07:15:22.766427 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:22.766346 4951 scope.go:122] "RemoveContainer" containerID="77efbacf706942ea3720dd23cc33586e01f5e3e20a0fe69bc0313919601a68a0" Oct 30 07:15:22.766716 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:22.766685 4951 scope.go:122] "RemoveContainer" containerID="05da206ee2f8305babc4cdf8a1e89e5ef0a38add4a00a0c5512114175d71dbe9" Oct 30 07:15:22.766963 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:15:22.766940 4951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq_odh-ai-gateway-infra(bfd1f8d4-cc32-47f5-9f87-954c79e326e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq" podUID="bfd1f8d4-cc32-47f5-9f87-954c79e326e3" Oct 30 07:15:22.814347 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:22.814315 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq"] Oct 30 07:15:23.025329 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:15:23.025299 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:15:23.776556 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:23.776524 4951 generic.go:356] "Generic (PLEG): container finished" podID="06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1" containerID="d82a93012b42ca142a88d777a57c0aef3cb2fc9661eea2ee0d5ef45763968f5e" exitCode=60 Oct 30 07:15:23.777076 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:23.776597 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq" event={"ID":"06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1","Type":"ContainerDied","Data":"d82a93012b42ca142a88d777a57c0aef3cb2fc9661eea2ee0d5ef45763968f5e"} Oct 30 07:15:23.777076 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:23.776653 4951 scope.go:122] "RemoveContainer" containerID="141be094bb75870103f1860ab11ef397af0f01eaf09821f4ceac557476eb060c" Oct 30 07:15:23.777076 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:23.777060 4951 scope.go:122] "RemoveContainer" containerID="d82a93012b42ca142a88d777a57c0aef3cb2fc9661eea2ee0d5ef45763968f5e" Oct 30 07:15:23.777445 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:15:23.777421 4951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq_odh-ai-gateway-infra(06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq" podUID="06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1" Oct 30 07:15:23.962761 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:23.962736 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq" Oct 30 07:15:24.104409 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:24.104317 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfd1f8d4-cc32-47f5-9f87-954c79e326e3-kube-api-access-vgbqf\" (UniqueName: \"kubernetes.io/projected/bfd1f8d4-cc32-47f5-9f87-954c79e326e3-kube-api-access-vgbqf\") pod \"bfd1f8d4-cc32-47f5-9f87-954c79e326e3\" (UID: \"bfd1f8d4-cc32-47f5-9f87-954c79e326e3\") " Oct 30 07:15:24.104625 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:24.104541 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfd1f8d4-cc32-47f5-9f87-954c79e326e3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfd1f8d4-cc32-47f5-9f87-954c79e326e3-maas-api-service-ca\") pod \"bfd1f8d4-cc32-47f5-9f87-954c79e326e3\" (UID: \"bfd1f8d4-cc32-47f5-9f87-954c79e326e3\") " Oct 30 07:15:24.104912 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:24.104891 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfd1f8d4-cc32-47f5-9f87-954c79e326e3-maas-api-service-ca" pod "bfd1f8d4-cc32-47f5-9f87-954c79e326e3" (UID: "bfd1f8d4-cc32-47f5-9f87-954c79e326e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:15:24.105168 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:24.105154 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfd1f8d4-cc32-47f5-9f87-954c79e326e3-maas-api-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:15:24.106518 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:24.106497 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd1f8d4-cc32-47f5-9f87-954c79e326e3-kube-api-access-vgbqf" pod "bfd1f8d4-cc32-47f5-9f87-954c79e326e3" (UID: "bfd1f8d4-cc32-47f5-9f87-954c79e326e3"). InnerVolumeSpecName "kube-api-access-vgbqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:15:24.206773 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:24.206741 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgbqf\" (UniqueName: \"kubernetes.io/projected/bfd1f8d4-cc32-47f5-9f87-954c79e326e3-kube-api-access-vgbqf\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:15:24.807501 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:24.807452 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq" event={"ID":"bfd1f8d4-cc32-47f5-9f87-954c79e326e3","Type":"ContainerDied","Data":"35cef0110dd51b1ae592155a7d8601fa699fa599bad978d3ccaef4088ca0a3f0"} Oct 30 07:15:24.808518 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:24.808494 4951 scope.go:122] "RemoveContainer" containerID="05da206ee2f8305babc4cdf8a1e89e5ef0a38add4a00a0c5512114175d71dbe9" Oct 30 07:15:24.809883 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:24.808404 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq" Oct 30 07:15:24.845647 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:24.845614 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq"] Oct 30 07:15:24.930363 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:24.930323 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq"] Oct 30 07:15:24.936081 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:24.936049 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889075-hbkgq"] Oct 30 07:15:25.053980 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:25.053950 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq" Oct 30 07:15:25.231152 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:25.231058 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1-kube-api-access-pmwsg\" (UniqueName: \"kubernetes.io/projected/06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1-kube-api-access-pmwsg\") pod \"06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1\" (UID: \"06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1\") " Oct 30 07:15:25.231152 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:25.231105 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1-maas-api-service-ca\") pod \"06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1\" (UID: \"06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1\") " Oct 30 07:15:25.231662 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:25.231632 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1-maas-api-service-ca" pod "06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1" (UID: "06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:15:25.231982 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:25.231960 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1-maas-api-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:15:25.233481 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:25.233452 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1-kube-api-access-pmwsg" pod "06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1" (UID: "06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1"). InnerVolumeSpecName "kube-api-access-pmwsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:15:25.333113 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:25.333074 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmwsg\" (UniqueName: \"kubernetes.io/projected/06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1-kube-api-access-pmwsg\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:15:25.773756 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:25.773721 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfd1f8d4-cc32-47f5-9f87-954c79e326e3" path="/var/lib/kubelet/pods/bfd1f8d4-cc32-47f5-9f87-954c79e326e3/volumes" Oct 30 07:15:25.845143 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:25.845106 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq" Oct 30 07:15:25.845558 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:25.845141 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq" event={"ID":"06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1","Type":"ContainerDied","Data":"cb08e619c7e0dd735292a89aa448b79383f7e53414088ec8a77231ce33d01320"} Oct 30 07:15:25.845558 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:25.845198 4951 scope.go:122] "RemoveContainer" containerID="d82a93012b42ca142a88d777a57c0aef3cb2fc9661eea2ee0d5ef45763968f5e" Oct 30 07:15:25.896367 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:25.896334 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq"] Oct 30 07:15:25.902536 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:25.902506 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889075-wmdnq"] Oct 30 07:15:27.772360 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:27.772330 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1" path="/var/lib/kubelet/pods/06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1/volumes" Oct 30 07:15:30.415775 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.415739 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2"] Oct 30 07:15:30.416859 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.416699 4951 cpu_manager.go:414] "containerMap: removing container" podUID="06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1" containerName="cleanup" Oct 30 07:15:30.416859 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.416720 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1" containerName="cleanup" Oct 30 07:15:30.416859 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.416736 4951 cpu_manager.go:414] "containerMap: removing container" podUID="bfd1f8d4-cc32-47f5-9f87-954c79e326e3" containerName="cleanup" Oct 30 07:15:30.416859 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.416744 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfd1f8d4-cc32-47f5-9f87-954c79e326e3" containerName="cleanup" Oct 30 07:15:30.416859 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.416804 4951 cpu_manager.go:414] "containerMap: removing container" podUID="06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1" containerName="cleanup" Oct 30 07:15:30.416859 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.416811 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1" containerName="cleanup" Oct 30 07:15:30.416859 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.416846 4951 cpu_manager.go:414] "containerMap: removing container" podUID="fc5c70db-419b-439d-aa07-a9bdfc78aad0" containerName="revoke-keys" Oct 30 07:15:30.416859 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.416854 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc5c70db-419b-439d-aa07-a9bdfc78aad0" containerName="revoke-keys" Oct 30 07:15:30.417264 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.416870 4951 cpu_manager.go:414] "containerMap: removing container" podUID="bfd1f8d4-cc32-47f5-9f87-954c79e326e3" containerName="cleanup" Oct 30 07:15:30.417264 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.416878 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfd1f8d4-cc32-47f5-9f87-954c79e326e3" containerName="cleanup" Oct 30 07:15:30.417264 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.417017 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1" containerName="cleanup" Oct 30 07:15:30.417264 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.417031 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1" containerName="cleanup" Oct 30 07:15:30.417264 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.417047 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfd1f8d4-cc32-47f5-9f87-954c79e326e3" containerName="cleanup" Oct 30 07:15:30.417264 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.417055 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc5c70db-419b-439d-aa07-a9bdfc78aad0" containerName="revoke-keys" Oct 30 07:15:30.417264 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.417064 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfd1f8d4-cc32-47f5-9f87-954c79e326e3" containerName="cleanup" Oct 30 07:15:30.417264 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.417073 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfd1f8d4-cc32-47f5-9f87-954c79e326e3" containerName="cleanup" Oct 30 07:15:30.422941 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.422915 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:15:30.439204 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.439175 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-77ef52-openshift-default-dockercfg-vf7lq\"" Oct 30 07:15:30.454303 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.454267 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2"] Oct 30 07:15:30.531996 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.531965 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnsxv\" (UniqueName: \"kubernetes.io/projected/8a282e3a-aa71-4b0a-b9db-f475645f94e6-kube-api-access-gnsxv\") pod \"e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:15:30.531996 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.532006 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istio-data\") pod \"e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:15:30.532215 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.532193 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istio-envoy\") pod \"e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:15:30.532272 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.532226 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istio-token\") pod \"e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:15:30.532272 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.532265 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istio-podinfo\") pod \"e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:15:30.532402 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.532388 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-workload-certs\") pod \"e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:15:30.532440 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.532434 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-workload-socket\") pod \"e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:15:30.532494 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.532450 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istiod-ca-cert\") pod \"e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:15:30.532636 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.532502 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-credential-socket\") pod \"e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:15:30.637642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.634948 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istio-envoy\") pod \"e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:15:30.637642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.635028 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istio-token\") pod \"e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:15:30.637642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.635084 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istio-podinfo\") pod \"e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:15:30.637642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.635141 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-workload-certs\") pod \"e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:15:30.637642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.635217 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-workload-socket\") pod \"e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:15:30.637642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.635249 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istiod-ca-cert\") pod \"e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:15:30.637642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.635277 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-credential-socket\") pod \"e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:15:30.637642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.635343 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnsxv\" (UniqueName: \"kubernetes.io/projected/8a282e3a-aa71-4b0a-b9db-f475645f94e6-kube-api-access-gnsxv\") pod \"e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:15:30.637642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.635382 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istio-data\") pod \"e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:15:30.637642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.636053 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istio-data\") pod \"e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:15:30.648594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.640909 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-workload-socket\") pod \"e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:15:30.648594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.646984 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-credential-socket\") pod \"e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:15:30.648594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.647671 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istiod-ca-cert\") pod \"e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:15:30.648594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.648204 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-workload-certs\") pod \"e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:15:30.655594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.653177 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istio-podinfo\") pod \"e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:15:30.699259 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.699226 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istio-token\") pod \"e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:15:30.706635 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.700584 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istio-envoy\") pod \"e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:15:30.707498 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.707469 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnsxv\" (UniqueName: \"kubernetes.io/projected/8a282e3a-aa71-4b0a-b9db-f475645f94e6-kube-api-access-gnsxv\") pod \"e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:15:30.751595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:30.751192 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:15:31.321338 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:31.321296 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2"] Oct 30 07:15:31.322183 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:15:31.322156 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a282e3a_aa71_4b0a_b9db_f475645f94e6.slice/crio-e19be68153f4ae36d3d33f98d0478eabdf97c72963da40d162b1f3a24b8048b8 WatchSource:0}: Error finding container e19be68153f4ae36d3d33f98d0478eabdf97c72963da40d162b1f3a24b8048b8: Status 404 returned error can't find the container with id e19be68153f4ae36d3d33f98d0478eabdf97c72963da40d162b1f3a24b8048b8 Oct 30 07:15:31.324614 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:31.324545 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:15:31.324699 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:31.324653 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:15:31.324699 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:31.324687 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:15:31.951339 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:31.951299 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" event={"ID":"8a282e3a-aa71-4b0a-b9db-f475645f94e6","Type":"ContainerStarted","Data":"993ee6f9224afbd09b035fb21c698416775eddccf8f94a896ef1e155d191c172"} Oct 30 07:15:31.951862 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:31.951385 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" event={"ID":"8a282e3a-aa71-4b0a-b9db-f475645f94e6","Type":"ContainerStarted","Data":"e19be68153f4ae36d3d33f98d0478eabdf97c72963da40d162b1f3a24b8048b8"} Oct 30 07:15:32.005919 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:32.005861 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" podStartSLOduration=2.005845133 podStartE2EDuration="2.005845133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:32.004173461 +0000 UTC m=+1138.347221426" watchObservedRunningTime="2026-10-30 07:15:32.005845133 +0000 UTC m=+1138.348893099" Oct 30 07:15:32.767594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:32.760220 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:15:32.807600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:32.807328 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:15:32.958033 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:32.957990 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:15:32.959081 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:32.959056 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:15:34.678751 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.678703 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc"] Oct 30 07:15:34.679840 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.679811 4951 cpu_manager.go:414] "containerMap: removing container" podUID="06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1" containerName="cleanup" Oct 30 07:15:34.679840 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.679839 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1" containerName="cleanup" Oct 30 07:15:34.680060 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.679856 4951 cpu_manager.go:414] "containerMap: removing container" podUID="bfd1f8d4-cc32-47f5-9f87-954c79e326e3" containerName="cleanup" Oct 30 07:15:34.680060 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.679864 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfd1f8d4-cc32-47f5-9f87-954c79e326e3" containerName="cleanup" Oct 30 07:15:34.680171 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.680094 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="06a9e4f0-da4c-4abc-a5d1-fcfa4a6bb2a1" containerName="cleanup" Oct 30 07:15:34.686430 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.686400 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:15:34.698002 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.697965 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d36f333-0885-434c-aea5-0f9f1d44217c-istio-podinfo\") pod \"e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:15:34.698189 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.698029 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d36f333-0885-434c-aea5-0f9f1d44217c-istio-token\") pod \"e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:15:34.698189 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.698062 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv26d\" (UniqueName: \"kubernetes.io/projected/2d36f333-0885-434c-aea5-0f9f1d44217c-kube-api-access-zv26d\") pod \"e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:15:34.698189 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.698118 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d36f333-0885-434c-aea5-0f9f1d44217c-istiod-ca-cert\") pod \"e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:15:34.698189 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.698172 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-workload-certs\") pod \"e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:15:34.698458 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.698206 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-istio-data\") pod \"e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:15:34.698458 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.698282 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-workload-socket\") pod \"e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:15:34.698458 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.698333 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-istio-envoy\") pod \"e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:15:34.698458 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.698429 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-credential-socket\") pod \"e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:15:34.700466 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.700435 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc"] Oct 30 07:15:34.706089 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.706048 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-7c51aa-openshift-default-dockercfg-fnzzg\"" Oct 30 07:15:34.799283 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.799241 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-istio-data\") pod \"e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:15:34.799459 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.799314 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-workload-socket\") pod \"e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:15:34.799459 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.799351 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-istio-envoy\") pod \"e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:15:34.799459 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.799422 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-credential-socket\") pod \"e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:15:34.799761 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.799475 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d36f333-0885-434c-aea5-0f9f1d44217c-istio-podinfo\") pod \"e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:15:34.799761 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.799520 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d36f333-0885-434c-aea5-0f9f1d44217c-istio-token\") pod \"e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:15:34.799761 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.799548 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv26d\" (UniqueName: \"kubernetes.io/projected/2d36f333-0885-434c-aea5-0f9f1d44217c-kube-api-access-zv26d\") pod \"e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:15:34.799761 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.799660 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d36f333-0885-434c-aea5-0f9f1d44217c-istiod-ca-cert\") pod \"e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:15:34.799761 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.799731 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-workload-certs\") pod \"e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:15:34.800034 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.799783 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-istio-data\") pod \"e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:15:34.800034 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.799805 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-workload-socket\") pod \"e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:15:34.800034 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.799872 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-credential-socket\") pod \"e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:15:34.800142 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.800064 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-workload-certs\") pod \"e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:15:34.800423 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.800397 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d36f333-0885-434c-aea5-0f9f1d44217c-istiod-ca-cert\") pod \"e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:15:34.801819 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.801801 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-istio-envoy\") pod \"e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:15:34.801950 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.801932 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d36f333-0885-434c-aea5-0f9f1d44217c-istio-podinfo\") pod \"e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:15:34.822235 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.822200 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d36f333-0885-434c-aea5-0f9f1d44217c-istio-token\") pod \"e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:15:34.822393 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:34.822316 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv26d\" (UniqueName: \"kubernetes.io/projected/2d36f333-0885-434c-aea5-0f9f1d44217c-kube-api-access-zv26d\") pod \"e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:15:35.011597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:35.011538 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:15:35.472518 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:15:35.472484 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d36f333_0885_434c_aea5_0f9f1d44217c.slice/crio-e92f95ffa81c6d241d87307eaf062aa9ccf5b2f645ea441c56beb1e21d471600 WatchSource:0}: Error finding container e92f95ffa81c6d241d87307eaf062aa9ccf5b2f645ea441c56beb1e21d471600: Status 404 returned error can't find the container with id e92f95ffa81c6d241d87307eaf062aa9ccf5b2f645ea441c56beb1e21d471600 Oct 30 07:15:35.473098 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:35.473074 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc"] Oct 30 07:15:35.475086 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:35.475052 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:15:35.475178 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:35.475116 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:15:35.475216 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:35.475176 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:15:36.095215 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:36.095178 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" event={"ID":"2d36f333-0885-434c-aea5-0f9f1d44217c","Type":"ContainerStarted","Data":"4946296448e1b6c3ff58d749bec9d611b7141e8fdc636827aeff859762826b71"} Oct 30 07:15:36.095633 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:36.095239 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" event={"ID":"2d36f333-0885-434c-aea5-0f9f1d44217c","Type":"ContainerStarted","Data":"e92f95ffa81c6d241d87307eaf062aa9ccf5b2f645ea441c56beb1e21d471600"} Oct 30 07:15:36.141511 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:36.141449 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" podStartSLOduration=2.141428823 podStartE2EDuration="2.141428823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:36.136515917 +0000 UTC m=+1142.479563885" watchObservedRunningTime="2026-10-30 07:15:36.141428823 +0000 UTC m=+1142.484476789" Oct 30 07:15:37.013123 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:37.013083 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:15:37.018670 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:37.018640 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:15:37.101991 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:37.101959 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:15:37.103088 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:37.103064 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:15:40.375217 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:40.375175 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7c51aa-5dd58b9fcf-q65zk"] Oct 30 07:15:40.385650 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:40.385621 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c51aa-5dd58b9fcf-q65zk" Oct 30 07:15:40.400593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:40.400545 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7c51aa-5dd58b9fcf-q65zk"] Oct 30 07:15:40.405810 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:40.405781 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-7c51aa\"" Oct 30 07:15:40.405963 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:40.405895 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-7c51aa-dockercfg-ckdhg\"" Oct 30 07:15:40.467275 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:40.467225 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgv6b\" (UniqueName: \"kubernetes.io/projected/10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4-kube-api-access-zgv6b\") pod \"payload-pre-processing-e2e-api-a-7c51aa-5dd58b9fcf-q65zk\" (UID: \"10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c51aa-5dd58b9fcf-q65zk" Oct 30 07:15:40.467863 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:40.467829 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7c51aa-5dd58b9fcf-q65zk\" (UID: \"10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c51aa-5dd58b9fcf-q65zk" Oct 30 07:15:40.513596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:40.513297 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7c51aa-784f94685d-469jj"] Oct 30 07:15:40.587842 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:40.587782 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7c51aa-5dd58b9fcf-q65zk\" (UID: \"10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c51aa-5dd58b9fcf-q65zk" Oct 30 07:15:40.588434 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:40.588401 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgv6b\" (UniqueName: \"kubernetes.io/projected/10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4-kube-api-access-zgv6b\") pod \"payload-pre-processing-e2e-api-a-7c51aa-5dd58b9fcf-q65zk\" (UID: \"10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c51aa-5dd58b9fcf-q65zk" Oct 30 07:15:40.622591 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:40.589506 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7c51aa-5dd58b9fcf-q65zk\" (UID: \"10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c51aa-5dd58b9fcf-q65zk" Oct 30 07:15:40.622846 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:40.613022 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7c51aa-784f94685d-469jj" Oct 30 07:15:40.656983 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:40.656895 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7c51aa-784f94685d-469jj"] Oct 30 07:15:40.691765 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:40.691711 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfhf6\" (UniqueName: \"kubernetes.io/projected/5caded86-2432-42eb-8179-f88d95e73857-kube-api-access-jfhf6\") pod \"payload-processing-e2e-api-a-7c51aa-784f94685d-469jj\" (UID: \"5caded86-2432-42eb-8179-f88d95e73857\") " pod="openshift-ingress/payload-processing-e2e-api-a-7c51aa-784f94685d-469jj" Oct 30 07:15:40.692059 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:40.692035 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5caded86-2432-42eb-8179-f88d95e73857-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7c51aa-784f94685d-469jj\" (UID: \"5caded86-2432-42eb-8179-f88d95e73857\") " pod="openshift-ingress/payload-processing-e2e-api-a-7c51aa-784f94685d-469jj" Oct 30 07:15:40.700235 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:40.700194 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgv6b\" (UniqueName: \"kubernetes.io/projected/10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4-kube-api-access-zgv6b\") pod \"payload-pre-processing-e2e-api-a-7c51aa-5dd58b9fcf-q65zk\" (UID: \"10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c51aa-5dd58b9fcf-q65zk" Oct 30 07:15:40.718325 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:40.718281 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c51aa-5dd58b9fcf-q65zk" Oct 30 07:15:40.822908 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:40.806960 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfhf6\" (UniqueName: \"kubernetes.io/projected/5caded86-2432-42eb-8179-f88d95e73857-kube-api-access-jfhf6\") pod \"payload-processing-e2e-api-a-7c51aa-784f94685d-469jj\" (UID: \"5caded86-2432-42eb-8179-f88d95e73857\") " pod="openshift-ingress/payload-processing-e2e-api-a-7c51aa-784f94685d-469jj" Oct 30 07:15:40.822908 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:40.818070 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5caded86-2432-42eb-8179-f88d95e73857-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7c51aa-784f94685d-469jj\" (UID: \"5caded86-2432-42eb-8179-f88d95e73857\") " pod="openshift-ingress/payload-processing-e2e-api-a-7c51aa-784f94685d-469jj" Oct 30 07:15:40.822908 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:40.822266 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5caded86-2432-42eb-8179-f88d95e73857-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7c51aa-784f94685d-469jj\" (UID: \"5caded86-2432-42eb-8179-f88d95e73857\") " pod="openshift-ingress/payload-processing-e2e-api-a-7c51aa-784f94685d-469jj" Oct 30 07:15:40.839458 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:40.839421 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfhf6\" (UniqueName: \"kubernetes.io/projected/5caded86-2432-42eb-8179-f88d95e73857-kube-api-access-jfhf6\") pod \"payload-processing-e2e-api-a-7c51aa-784f94685d-469jj\" (UID: \"5caded86-2432-42eb-8179-f88d95e73857\") " pod="openshift-ingress/payload-processing-e2e-api-a-7c51aa-784f94685d-469jj" Oct 30 07:15:41.069326 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:41.069294 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7c51aa-784f94685d-469jj" Oct 30 07:15:41.177887 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:15:41.177856 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a59bb2_4606_4c0a_baf0_0d5fbabc1ba4.slice/crio-c0d575676091d5ffabb1ea9c83bbea18481834578a767216ec33d6f882d9684d WatchSource:0}: Error finding container c0d575676091d5ffabb1ea9c83bbea18481834578a767216ec33d6f882d9684d: Status 404 returned error can't find the container with id c0d575676091d5ffabb1ea9c83bbea18481834578a767216ec33d6f882d9684d Oct 30 07:15:41.177887 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:41.177867 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7c51aa-5dd58b9fcf-q65zk"] Oct 30 07:15:41.373361 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:15:41.373326 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5caded86_2432_42eb_8179_f88d95e73857.slice/crio-80d0b0c8bdc82850981e6f163cda13d8d0452c06ad2008052e7cbf22dcf0acd6 WatchSource:0}: Error finding container 80d0b0c8bdc82850981e6f163cda13d8d0452c06ad2008052e7cbf22dcf0acd6: Status 404 returned error can't find the container with id 80d0b0c8bdc82850981e6f163cda13d8d0452c06ad2008052e7cbf22dcf0acd6 Oct 30 07:15:41.374047 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:41.374025 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7c51aa-784f94685d-469jj"] Oct 30 07:15:42.153295 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:42.153240 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7c51aa-784f94685d-469jj" event={"ID":"5caded86-2432-42eb-8179-f88d95e73857","Type":"ContainerStarted","Data":"0b1d7ab2ad79b1df04161a75428f0f3b0766ff67f8adaadced8817111b357c57"} Oct 30 07:15:42.153838 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:42.153321 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-7c51aa-784f94685d-469jj" Oct 30 07:15:42.153838 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:42.153357 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7c51aa-784f94685d-469jj" event={"ID":"5caded86-2432-42eb-8179-f88d95e73857","Type":"ContainerStarted","Data":"80d0b0c8bdc82850981e6f163cda13d8d0452c06ad2008052e7cbf22dcf0acd6"} Oct 30 07:15:42.155080 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:42.155051 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c51aa-5dd58b9fcf-q65zk" event={"ID":"10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4","Type":"ContainerStarted","Data":"f57180a9c3a256d37db525447c1dfc9b4a3991955d851d28329acc93506de23b"} Oct 30 07:15:42.155214 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:42.155093 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c51aa-5dd58b9fcf-q65zk" event={"ID":"10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4","Type":"ContainerStarted","Data":"c0d575676091d5ffabb1ea9c83bbea18481834578a767216ec33d6f882d9684d"} Oct 30 07:15:42.155214 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:42.155137 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c51aa-5dd58b9fcf-q65zk" Oct 30 07:15:42.195991 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:42.195938 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-7c51aa-784f94685d-469jj" podStartSLOduration=2.19592346 podStartE2EDuration="2.19592346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:42.194527005 +0000 UTC m=+1148.537574974" watchObservedRunningTime="2026-10-30 07:15:42.19592346 +0000 UTC m=+1148.538971425" Oct 30 07:15:42.240777 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:42.240718 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c51aa-5dd58b9fcf-q65zk" podStartSLOduration=2.2407014419999998 podStartE2EDuration="2.240701442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:42.238651904 +0000 UTC m=+1148.581699883" watchObservedRunningTime="2026-10-30 07:15:42.240701442 +0000 UTC m=+1148.583749406" Oct 30 07:15:43.702547 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:43.702511 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8"] Oct 30 07:15:43.706413 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:43.706390 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:15:43.733097 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:43.733055 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8"] Oct 30 07:15:43.849066 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:43.847540 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:15:43.849066 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:43.847698 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:15:43.849066 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:43.847733 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:15:43.849066 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:43.847769 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:15:43.849066 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:43.847822 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:15:43.849066 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:43.847859 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:15:43.849066 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:43.847911 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:15:43.849066 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:43.847971 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6fwv\" (UniqueName: \"kubernetes.io/projected/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-kube-api-access-n6fwv\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:15:43.849066 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:43.848091 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:15:43.949676 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:43.949628 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:15:43.949881 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:43.949683 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:15:43.949881 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:43.949720 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:15:43.949881 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:43.949772 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:15:43.949881 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:43.949806 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:15:43.949881 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:43.949854 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:15:43.950076 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:43.949906 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6fwv\" (UniqueName: \"kubernetes.io/projected/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-kube-api-access-n6fwv\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:15:43.950076 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:43.950011 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:15:43.950169 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:43.950110 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:15:43.950527 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:43.950493 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:15:43.950690 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:43.950564 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:15:43.951018 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:43.950994 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:15:43.951126 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:43.951025 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:15:43.951126 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:43.951102 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:15:43.952965 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:43.952912 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:15:43.953056 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:43.952978 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:15:43.972949 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:43.972908 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:15:43.972949 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:43.972943 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6fwv\" (UniqueName: \"kubernetes.io/projected/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-kube-api-access-n6fwv\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:15:44.023154 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:44.023121 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:15:44.345402 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:15:44.345360 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d9a6b57_897d_4c14_88d7_87190ce5c8b8.slice/crio-1b9cc5f9920c6924bb3a194ceee9c6393961f6a5f67f1c3c966d6cadc3db197a WatchSource:0}: Error finding container 1b9cc5f9920c6924bb3a194ceee9c6393961f6a5f67f1c3c966d6cadc3db197a: Status 404 returned error can't find the container with id 1b9cc5f9920c6924bb3a194ceee9c6393961f6a5f67f1c3c966d6cadc3db197a Oct 30 07:15:44.346017 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:44.345989 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8"] Oct 30 07:15:44.349737 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:44.349698 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:15:44.349837 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:44.349774 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:15:44.349837 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:44.349805 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:15:45.217592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:45.209866 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" event={"ID":"4d9a6b57-897d-4c14-88d7-87190ce5c8b8","Type":"ContainerStarted","Data":"17e28c0df13c90588332a54e453eed4d487c1e2fd222074d79b2eadde214894e"} Oct 30 07:15:45.217592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:45.209945 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" event={"ID":"4d9a6b57-897d-4c14-88d7-87190ce5c8b8","Type":"ContainerStarted","Data":"1b9cc5f9920c6924bb3a194ceee9c6393961f6a5f67f1c3c966d6cadc3db197a"} Oct 30 07:15:45.280672 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:45.280605 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" podStartSLOduration=2.280555688 podStartE2EDuration="2.280555688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:45.278537765 +0000 UTC m=+1151.621585750" watchObservedRunningTime="2026-10-30 07:15:45.280555688 +0000 UTC m=+1151.623603652" Oct 30 07:15:46.033702 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:46.033660 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:15:46.033922 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:46.033793 4951 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 07:15:46.033922 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:46.033840 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" podUID="4d9a6b57-897d-4c14-88d7-87190ce5c8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 07:15:47.048594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:47.048017 4951 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 07:15:47.048594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:47.048136 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" podUID="4d9a6b57-897d-4c14-88d7-87190ce5c8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 07:15:48.024606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:48.023990 4951 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 07:15:48.024606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:48.024055 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" podUID="4d9a6b57-897d-4c14-88d7-87190ce5c8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 07:15:49.040060 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:49.040017 4951 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 07:15:49.040713 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:49.040678 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" podUID="4d9a6b57-897d-4c14-88d7-87190ce5c8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 07:15:50.030596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:50.028937 4951 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 07:15:50.030596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:50.029002 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" podUID="4d9a6b57-897d-4c14-88d7-87190ce5c8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 07:15:51.023808 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:51.023773 4951 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 07:15:51.024205 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:51.023829 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" podUID="4d9a6b57-897d-4c14-88d7-87190ce5c8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 07:15:52.370926 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:52.370890 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:15:52.371417 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:52.371397 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:15:52.372130 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:52.372108 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:15:53.056993 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:15:53.056959 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:15:53.227903 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:53.227863 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c51aa-5dd58b9fcf-q65zk" Oct 30 07:15:53.229661 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:53.229633 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-7c51aa-784f94685d-469jj" Oct 30 07:15:56.442701 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:56.437181 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-9bfvf"] Oct 30 07:15:56.457805 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:56.457772 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-9bfvf"] Oct 30 07:15:56.464598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:56.458111 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-9bfvf" Oct 30 07:15:56.488244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:56.488211 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d0dace8c\"" Oct 30 07:15:56.496022 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:56.495990 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d0dace8c-dockercfg-7wqnw\"" Oct 30 07:15:56.527797 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:56.527434 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljzs4\" (UniqueName: \"kubernetes.io/projected/8bc553e5-f6d7-49e4-b4e1-d26f35e53cad-kube-api-access-ljzs4\") pod \"payload-processing-e2e-mt-d0dace8c-5964688d84-9bfvf\" (UID: \"8bc553e5-f6d7-49e4-b4e1-d26f35e53cad\") " pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-9bfvf" Oct 30 07:15:56.527797 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:56.527506 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bc553e5-f6d7-49e4-b4e1-d26f35e53cad-plugins-config-volume\") pod \"payload-processing-e2e-mt-d0dace8c-5964688d84-9bfvf\" (UID: \"8bc553e5-f6d7-49e4-b4e1-d26f35e53cad\") " pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-9bfvf" Oct 30 07:15:56.660542 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:56.648025 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bc553e5-f6d7-49e4-b4e1-d26f35e53cad-plugins-config-volume\") pod \"payload-processing-e2e-mt-d0dace8c-5964688d84-9bfvf\" (UID: \"8bc553e5-f6d7-49e4-b4e1-d26f35e53cad\") " pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-9bfvf" Oct 30 07:15:56.702251 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:56.702158 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljzs4\" (UniqueName: \"kubernetes.io/projected/8bc553e5-f6d7-49e4-b4e1-d26f35e53cad-kube-api-access-ljzs4\") pod \"payload-processing-e2e-mt-d0dace8c-5964688d84-9bfvf\" (UID: \"8bc553e5-f6d7-49e4-b4e1-d26f35e53cad\") " pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-9bfvf" Oct 30 07:15:56.702968 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:56.699956 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bc553e5-f6d7-49e4-b4e1-d26f35e53cad-plugins-config-volume\") pod \"payload-processing-e2e-mt-d0dace8c-5964688d84-9bfvf\" (UID: \"8bc553e5-f6d7-49e4-b4e1-d26f35e53cad\") " pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-9bfvf" Oct 30 07:15:56.844704 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:56.844589 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljzs4\" (UniqueName: \"kubernetes.io/projected/8bc553e5-f6d7-49e4-b4e1-d26f35e53cad-kube-api-access-ljzs4\") pod \"payload-processing-e2e-mt-d0dace8c-5964688d84-9bfvf\" (UID: \"8bc553e5-f6d7-49e4-b4e1-d26f35e53cad\") " pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-9bfvf" Oct 30 07:15:57.132973 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:57.132935 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-9bfvf" Oct 30 07:15:57.436812 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:57.436781 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-9bfvf"] Oct 30 07:15:57.438100 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:15:57.438070 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bc553e5_f6d7_49e4_b4e1_d26f35e53cad.slice/crio-2277c7166a865e41f0ec759815c8f887ec29bf4d781bf4671fe6a403da3e0d1a WatchSource:0}: Error finding container 2277c7166a865e41f0ec759815c8f887ec29bf4d781bf4671fe6a403da3e0d1a: Status 404 returned error can't find the container with id 2277c7166a865e41f0ec759815c8f887ec29bf4d781bf4671fe6a403da3e0d1a Oct 30 07:15:57.452916 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:57.452791 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-9bfvf" event={"ID":"8bc553e5-f6d7-49e4-b4e1-d26f35e53cad","Type":"ContainerStarted","Data":"2277c7166a865e41f0ec759815c8f887ec29bf4d781bf4671fe6a403da3e0d1a"} Oct 30 07:15:58.096312 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:58.096283 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl"] Oct 30 07:15:58.100668 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:58.100648 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl" Oct 30 07:15:58.124654 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:58.124617 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d0dace8c-serving-cert\"" Oct 30 07:15:58.132977 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:58.132940 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl"] Oct 30 07:15:58.155562 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:58.155530 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n25t\" (UniqueName: \"kubernetes.io/projected/2b9ab9fc-ba02-4743-9105-0140368dd943-kube-api-access-5n25t\") pod \"maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl\" (UID: \"2b9ab9fc-ba02-4743-9105-0140368dd943\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl" Oct 30 07:15:58.155764 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:58.155619 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b9ab9fc-ba02-4743-9105-0140368dd943-maas-api-tls\") pod \"maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl\" (UID: \"2b9ab9fc-ba02-4743-9105-0140368dd943\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl" Oct 30 07:15:58.155840 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:58.155804 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b9ab9fc-ba02-4743-9105-0140368dd943-openshift-service-ca\") pod \"maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl\" (UID: \"2b9ab9fc-ba02-4743-9105-0140368dd943\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl" Oct 30 07:15:58.155940 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:58.155919 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b9ab9fc-ba02-4743-9105-0140368dd943-metrics-certs\") pod \"maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl\" (UID: \"2b9ab9fc-ba02-4743-9105-0140368dd943\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl" Oct 30 07:15:58.257462 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:58.257422 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b9ab9fc-ba02-4743-9105-0140368dd943-openshift-service-ca\") pod \"maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl\" (UID: \"2b9ab9fc-ba02-4743-9105-0140368dd943\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl" Oct 30 07:15:58.257677 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:58.257517 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b9ab9fc-ba02-4743-9105-0140368dd943-metrics-certs\") pod \"maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl\" (UID: \"2b9ab9fc-ba02-4743-9105-0140368dd943\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl" Oct 30 07:15:58.257763 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:58.257684 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n25t\" (UniqueName: \"kubernetes.io/projected/2b9ab9fc-ba02-4743-9105-0140368dd943-kube-api-access-5n25t\") pod \"maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl\" (UID: \"2b9ab9fc-ba02-4743-9105-0140368dd943\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl" Oct 30 07:15:58.257763 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:58.257750 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b9ab9fc-ba02-4743-9105-0140368dd943-maas-api-tls\") pod \"maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl\" (UID: \"2b9ab9fc-ba02-4743-9105-0140368dd943\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl" Oct 30 07:15:58.258278 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:58.258238 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b9ab9fc-ba02-4743-9105-0140368dd943-openshift-service-ca\") pod \"maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl\" (UID: \"2b9ab9fc-ba02-4743-9105-0140368dd943\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl" Oct 30 07:15:58.260183 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:58.260150 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b9ab9fc-ba02-4743-9105-0140368dd943-metrics-certs\") pod \"maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl\" (UID: \"2b9ab9fc-ba02-4743-9105-0140368dd943\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl" Oct 30 07:15:58.260288 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:58.260259 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b9ab9fc-ba02-4743-9105-0140368dd943-maas-api-tls\") pod \"maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl\" (UID: \"2b9ab9fc-ba02-4743-9105-0140368dd943\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl" Oct 30 07:15:58.280055 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:58.280023 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n25t\" (UniqueName: \"kubernetes.io/projected/2b9ab9fc-ba02-4743-9105-0140368dd943-kube-api-access-5n25t\") pod \"maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl\" (UID: \"2b9ab9fc-ba02-4743-9105-0140368dd943\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl" Oct 30 07:15:58.422431 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:58.420064 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl" Oct 30 07:15:58.524120 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:58.524082 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-9bfvf" event={"ID":"8bc553e5-f6d7-49e4-b4e1-d26f35e53cad","Type":"ContainerStarted","Data":"d6d0a32cdbb5aa3e7092fd14f3a001aa894a11eb341c5be3a8f3a5c0c6019eb4"} Oct 30 07:15:58.525968 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:58.525941 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-9bfvf" Oct 30 07:15:58.569356 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:58.569303 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-9bfvf" podStartSLOduration=2.569289849 podStartE2EDuration="2.569289849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:58.563227377 +0000 UTC m=+1164.906275345" watchObservedRunningTime="2026-10-30 07:15:58.569289849 +0000 UTC m=+1164.912337814" Oct 30 07:15:58.852652 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:58.852619 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl"] Oct 30 07:15:58.853375 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:15:58.853341 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b9ab9fc_ba02_4743_9105_0140368dd943.slice/crio-eaced34667471c7b33824da6fa07046f900881f3c3bf9d3bfdb4da7849c6568a WatchSource:0}: Error finding container eaced34667471c7b33824da6fa07046f900881f3c3bf9d3bfdb4da7849c6568a: Status 404 returned error can't find the container with id eaced34667471c7b33824da6fa07046f900881f3c3bf9d3bfdb4da7849c6568a Oct 30 07:15:59.591983 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:59.591877 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl" event={"ID":"2b9ab9fc-ba02-4743-9105-0140368dd943","Type":"ContainerStarted","Data":"7aa33001bb4961850ac38f7c610f8ce5f3b6c148cca6fdaf40f7d1fc587301c2"} Oct 30 07:15:59.591983 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:59.591955 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl" event={"ID":"2b9ab9fc-ba02-4743-9105-0140368dd943","Type":"ContainerStarted","Data":"eaced34667471c7b33824da6fa07046f900881f3c3bf9d3bfdb4da7849c6568a"} Oct 30 07:15:59.592506 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:15:59.592008 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl" Oct 30 07:16:06.639652 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:06.638554 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl" Oct 30 07:16:06.685443 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:06.685393 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl" podStartSLOduration=8.241606143 podStartE2EDuration="8.685375481s" totalImagesPullingTime="443.769338ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:59.664876153 +0000 UTC m=+1166.007924163" watchObservedRunningTime="2026-10-30 07:16:06.685375481 +0000 UTC m=+1173.028423447" Oct 30 07:16:10.617597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:10.617281 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-9bfvf" Oct 30 07:16:12.539598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:12.538798 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1e3dd1-789c944fd4-m74hx"] Oct 30 07:16:12.665068 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:12.665018 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1e3dd1-789c944fd4-m74hx"] Oct 30 07:16:12.665465 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:12.665446 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1e3dd1-789c944fd4-m74hx" Oct 30 07:16:12.702611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:12.693212 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-1e3dd1-dockercfg-g4xnd\"" Oct 30 07:16:12.702611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:12.693343 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-1e3dd1\"" Oct 30 07:16:12.702611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:12.695028 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v98b\" (UniqueName: \"kubernetes.io/projected/fc1e4717-80e7-47d7-b7f7-75a9a2477d50-kube-api-access-7v98b\") pod \"payload-processing-e2e-api-b-1e3dd1-789c944fd4-m74hx\" (UID: \"fc1e4717-80e7-47d7-b7f7-75a9a2477d50\") " pod="openshift-ingress/payload-processing-e2e-api-b-1e3dd1-789c944fd4-m74hx" Oct 30 07:16:12.702611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:12.695094 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1e4717-80e7-47d7-b7f7-75a9a2477d50-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1e3dd1-789c944fd4-m74hx\" (UID: \"fc1e4717-80e7-47d7-b7f7-75a9a2477d50\") " pod="openshift-ingress/payload-processing-e2e-api-b-1e3dd1-789c944fd4-m74hx" Oct 30 07:16:12.798130 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:12.798030 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v98b\" (UniqueName: \"kubernetes.io/projected/fc1e4717-80e7-47d7-b7f7-75a9a2477d50-kube-api-access-7v98b\") pod \"payload-processing-e2e-api-b-1e3dd1-789c944fd4-m74hx\" (UID: \"fc1e4717-80e7-47d7-b7f7-75a9a2477d50\") " pod="openshift-ingress/payload-processing-e2e-api-b-1e3dd1-789c944fd4-m74hx" Oct 30 07:16:12.798387 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:12.798366 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1e4717-80e7-47d7-b7f7-75a9a2477d50-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1e3dd1-789c944fd4-m74hx\" (UID: \"fc1e4717-80e7-47d7-b7f7-75a9a2477d50\") " pod="openshift-ingress/payload-processing-e2e-api-b-1e3dd1-789c944fd4-m74hx" Oct 30 07:16:12.799616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:12.799592 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1e4717-80e7-47d7-b7f7-75a9a2477d50-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1e3dd1-789c944fd4-m74hx\" (UID: \"fc1e4717-80e7-47d7-b7f7-75a9a2477d50\") " pod="openshift-ingress/payload-processing-e2e-api-b-1e3dd1-789c944fd4-m74hx" Oct 30 07:16:13.074648 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:13.061655 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1e3dd1-695f9f69f9-w54sq"] Oct 30 07:16:13.111596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:13.109499 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v98b\" (UniqueName: \"kubernetes.io/projected/fc1e4717-80e7-47d7-b7f7-75a9a2477d50-kube-api-access-7v98b\") pod \"payload-processing-e2e-api-b-1e3dd1-789c944fd4-m74hx\" (UID: \"fc1e4717-80e7-47d7-b7f7-75a9a2477d50\") " pod="openshift-ingress/payload-processing-e2e-api-b-1e3dd1-789c944fd4-m74hx" Oct 30 07:16:13.168608 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:13.168556 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1e3dd1-695f9f69f9-w54sq"] Oct 30 07:16:13.169043 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:13.168951 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e3dd1-695f9f69f9-w54sq" Oct 30 07:16:13.179095 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:13.179072 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1e3dd1-789c944fd4-m74hx" Oct 30 07:16:13.339377 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:13.339278 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:13.352893 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:13.352855 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:13.393618 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:13.393562 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwj7c\" (UniqueName: \"kubernetes.io/projected/d2d13e7a-5d8a-417f-aff7-746e41ba4510-kube-api-access-jwj7c\") pod \"payload-pre-processing-e2e-api-b-1e3dd1-695f9f69f9-w54sq\" (UID: \"d2d13e7a-5d8a-417f-aff7-746e41ba4510\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e3dd1-695f9f69f9-w54sq" Oct 30 07:16:13.414595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:13.414540 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d13e7a-5d8a-417f-aff7-746e41ba4510-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1e3dd1-695f9f69f9-w54sq\" (UID: \"d2d13e7a-5d8a-417f-aff7-746e41ba4510\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e3dd1-695f9f69f9-w54sq" Oct 30 07:16:13.437952 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:13.437907 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 07:16:13.523631 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:13.523584 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwj7c\" (UniqueName: \"kubernetes.io/projected/d2d13e7a-5d8a-417f-aff7-746e41ba4510-kube-api-access-jwj7c\") pod \"payload-pre-processing-e2e-api-b-1e3dd1-695f9f69f9-w54sq\" (UID: \"d2d13e7a-5d8a-417f-aff7-746e41ba4510\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e3dd1-695f9f69f9-w54sq" Oct 30 07:16:13.524380 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:13.524353 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d13e7a-5d8a-417f-aff7-746e41ba4510-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1e3dd1-695f9f69f9-w54sq\" (UID: \"d2d13e7a-5d8a-417f-aff7-746e41ba4510\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e3dd1-695f9f69f9-w54sq" Oct 30 07:16:13.565594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:13.546646 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d13e7a-5d8a-417f-aff7-746e41ba4510-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1e3dd1-695f9f69f9-w54sq\" (UID: \"d2d13e7a-5d8a-417f-aff7-746e41ba4510\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e3dd1-695f9f69f9-w54sq" Oct 30 07:16:13.565594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:13.561374 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwj7c\" (UniqueName: \"kubernetes.io/projected/d2d13e7a-5d8a-417f-aff7-746e41ba4510-kube-api-access-jwj7c\") pod \"payload-pre-processing-e2e-api-b-1e3dd1-695f9f69f9-w54sq\" (UID: \"d2d13e7a-5d8a-417f-aff7-746e41ba4510\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e3dd1-695f9f69f9-w54sq" Oct 30 07:16:13.665671 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:13.647784 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e3dd1-695f9f69f9-w54sq" Oct 30 07:16:14.229595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:14.217103 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:14.254592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:14.250746 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 07:16:14.356607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:14.354688 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 07:16:14.380600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:14.379703 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:16:14.445550 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:14.426620 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:14.445550 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:14.427444 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:16:14.453529 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:14.452381 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 07:16:14.498901 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:14.498865 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:16:15.128644 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:15.097659 4951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:15.193488 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:15.153931 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:15.256598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:15.234567 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:15.256598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:15.235891 4951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:15.256598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:15.237547 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:16:15.274607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:15.265965 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:15.274607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:15.266366 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:16:15.361659 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:15.339908 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m8ksd" podUID="c9b6e389-2e4a-45d1-a3cf-a15d930ebb63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:15.407624 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:15.364686 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 07:16:15.485456 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:15.464916 4951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:15.687163 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:15.508727 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7dtn7" podUID="2cb568ba-79ac-405d-af09-bde6c2b47a53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:15.701014 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:15.700970 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 07:16:15.806689 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:15.806639 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:16.229856 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:16.229797 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 07:16:16.279726 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:16.279630 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 07:16:16.541326 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:16.541235 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:16.806250 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:16.806131 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:16.896395 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:16.878496 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:16.908878 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:16.908825 4951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded" Oct 30 07:16:16.941593 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:16:16.937824 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d13e7a_5d8a_417f_aff7_746e41ba4510.slice/crio-044434dfc32467f2d139cf6a807ccc40c76f3f27f59347512f12f439e80354ed WatchSource:0}: Error finding container 044434dfc32467f2d139cf6a807ccc40c76f3f27f59347512f12f439e80354ed: Status 404 returned error can't find the container with id 044434dfc32467f2d139cf6a807ccc40c76f3f27f59347512f12f439e80354ed Oct 30 07:16:17.468648 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:16:17.462328 4951 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.696s" Oct 30 07:16:17.468648 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:17.462442 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:16:17.468648 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:17.462478 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1e3dd1-789c944fd4-m74hx"] Oct 30 07:16:17.468648 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:17.462522 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1e3dd1-789c944fd4-m74hx" event={"ID":"fc1e4717-80e7-47d7-b7f7-75a9a2477d50","Type":"ContainerStarted","Data":"393d1623862f095403bb623de5c93890362643c853c1a8c518b7a94cd163f391"} Oct 30 07:16:17.468648 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:17.462613 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:16:17.468648 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:17.462650 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:16:17.468648 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:17.462681 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:16:17.564851 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:17.552022 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:17.623105 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:17.568654 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:17.623105 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:17.586712 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:16:17.809956 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:17.807846 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:16:17.815535 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:17.815385 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:16:17.834465 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:17.834437 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1e3dd1-695f9f69f9-w54sq"] Oct 30 07:16:17.889615 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:17.888899 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-56eb7f-5fbf45d446-d6gzv"] Oct 30 07:16:17.917642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:17.917480 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-56eb7f-5fbf45d446-d6gzv"] Oct 30 07:16:17.917642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:17.917607 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-56eb7f-5fbf45d446-d6gzv" Oct 30 07:16:17.983707 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:17.946940 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-56eb7f-dockercfg-wvvww\"" Oct 30 07:16:17.983707 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:17.947493 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-56eb7f\"" Oct 30 07:16:17.983707 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:17.948565 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glpmk\" (UniqueName: \"kubernetes.io/projected/1e764719-1b26-4ba1-87ba-fe5499e58071-kube-api-access-glpmk\") pod \"payload-processing-e2e-api-b-56eb7f-5fbf45d446-d6gzv\" (UID: \"1e764719-1b26-4ba1-87ba-fe5499e58071\") " pod="openshift-ingress/payload-processing-e2e-api-b-56eb7f-5fbf45d446-d6gzv" Oct 30 07:16:17.983707 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:17.948696 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e764719-1b26-4ba1-87ba-fe5499e58071-plugins-config-volume\") pod \"payload-processing-e2e-api-b-56eb7f-5fbf45d446-d6gzv\" (UID: \"1e764719-1b26-4ba1-87ba-fe5499e58071\") " pod="openshift-ingress/payload-processing-e2e-api-b-56eb7f-5fbf45d446-d6gzv" Oct 30 07:16:18.059642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:18.051668 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e764719-1b26-4ba1-87ba-fe5499e58071-plugins-config-volume\") pod \"payload-processing-e2e-api-b-56eb7f-5fbf45d446-d6gzv\" (UID: \"1e764719-1b26-4ba1-87ba-fe5499e58071\") " pod="openshift-ingress/payload-processing-e2e-api-b-56eb7f-5fbf45d446-d6gzv" Oct 30 07:16:18.059642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:18.052029 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glpmk\" (UniqueName: \"kubernetes.io/projected/1e764719-1b26-4ba1-87ba-fe5499e58071-kube-api-access-glpmk\") pod \"payload-processing-e2e-api-b-56eb7f-5fbf45d446-d6gzv\" (UID: \"1e764719-1b26-4ba1-87ba-fe5499e58071\") " pod="openshift-ingress/payload-processing-e2e-api-b-56eb7f-5fbf45d446-d6gzv" Oct 30 07:16:18.059642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:18.053438 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e764719-1b26-4ba1-87ba-fe5499e58071-plugins-config-volume\") pod \"payload-processing-e2e-api-b-56eb7f-5fbf45d446-d6gzv\" (UID: \"1e764719-1b26-4ba1-87ba-fe5499e58071\") " pod="openshift-ingress/payload-processing-e2e-api-b-56eb7f-5fbf45d446-d6gzv" Oct 30 07:16:18.339036 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:18.330834 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glpmk\" (UniqueName: \"kubernetes.io/projected/1e764719-1b26-4ba1-87ba-fe5499e58071-kube-api-access-glpmk\") pod \"payload-processing-e2e-api-b-56eb7f-5fbf45d446-d6gzv\" (UID: \"1e764719-1b26-4ba1-87ba-fe5499e58071\") " pod="openshift-ingress/payload-processing-e2e-api-b-56eb7f-5fbf45d446-d6gzv" Oct 30 07:16:18.421367 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:18.417274 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1e3dd1-789c944fd4-m74hx" event={"ID":"fc1e4717-80e7-47d7-b7f7-75a9a2477d50","Type":"ContainerStarted","Data":"10e1c51dda675c0a786c24efd613f4540a95953278a436468ff426c7ae51d938"} Oct 30 07:16:18.436592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:18.422823 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-1e3dd1-789c944fd4-m74hx" Oct 30 07:16:18.467592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:18.463404 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e3dd1-695f9f69f9-w54sq" event={"ID":"d2d13e7a-5d8a-417f-aff7-746e41ba4510","Type":"ContainerStarted","Data":"2055a75c086474ec00a0145cbe9d3c1470101879757142ced5357933e15f7d81"} Oct 30 07:16:18.467592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:18.463469 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e3dd1-695f9f69f9-w54sq" event={"ID":"d2d13e7a-5d8a-417f-aff7-746e41ba4510","Type":"ContainerStarted","Data":"044434dfc32467f2d139cf6a807ccc40c76f3f27f59347512f12f439e80354ed"} Oct 30 07:16:18.467592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:18.463497 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e3dd1-695f9f69f9-w54sq" Oct 30 07:16:18.504597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:18.471909 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-1e3dd1-789c944fd4-m74hx" podStartSLOduration=6.471886384 podStartE2EDuration="6.471886384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:16:18.464938868 +0000 UTC m=+1184.807986834" watchObservedRunningTime="2026-10-30 07:16:18.471886384 +0000 UTC m=+1184.814934349" Oct 30 07:16:18.597652 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:18.547356 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e3dd1-695f9f69f9-w54sq" podStartSLOduration=6.547332355 podStartE2EDuration="6.547332355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:16:18.510164238 +0000 UTC m=+1184.853212205" watchObservedRunningTime="2026-10-30 07:16:18.547332355 +0000 UTC m=+1184.890380321" Oct 30 07:16:18.610683 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:18.610651 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-56eb7f-5fbf45d446-d6gzv" Oct 30 07:16:19.297029 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:19.297000 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-56eb7f-5fbf45d446-d6gzv"] Oct 30 07:16:19.494739 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:19.494694 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-56eb7f-5fbf45d446-d6gzv" event={"ID":"1e764719-1b26-4ba1-87ba-fe5499e58071","Type":"ContainerStarted","Data":"3cad1a0b4b107577b15a9aca2b2893acbf6f65cd9b8ef562a973f70bc8d713fd"} Oct 30 07:16:19.494945 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:19.494759 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-56eb7f-5fbf45d446-d6gzv" event={"ID":"1e764719-1b26-4ba1-87ba-fe5499e58071","Type":"ContainerStarted","Data":"74f5828bdc4632591d3637acbe1e0a4f21029d67ff2df873af46915eefba8ba7"} Oct 30 07:16:19.495024 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:19.494952 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-56eb7f-5fbf45d446-d6gzv" Oct 30 07:16:23.029351 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:16:23.029309 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:16:23.143757 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:23.143699 4951 patch_prober.go:29] interesting pod/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:16:23.144023 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:23.143774 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" podUID="2d36f333-0885-434c-aea5-0f9f1d44217c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:24.540202 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:24.540170 4951 generic.go:356] "Generic (PLEG): container finished" podID="2d36f333-0885-434c-aea5-0f9f1d44217c" containerID="4946296448e1b6c3ff58d749bec9d611b7141e8fdc636827aeff859762826b71" exitCode=0 Oct 30 07:16:24.540782 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:24.540216 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" event={"ID":"2d36f333-0885-434c-aea5-0f9f1d44217c","Type":"ContainerDied","Data":"4946296448e1b6c3ff58d749bec9d611b7141e8fdc636827aeff859762826b71"} Oct 30 07:16:24.540866 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:24.540799 4951 scope.go:122] "RemoveContainer" containerID="4946296448e1b6c3ff58d749bec9d611b7141e8fdc636827aeff859762826b71" Oct 30 07:16:24.542545 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:24.542512 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:16:24.542682 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:24.542637 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:16:24.542754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:24.542692 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:16:24.606840 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:24.606115 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-56eb7f-5fbf45d446-d6gzv" podStartSLOduration=8.606096067 podStartE2EDuration="8.606096067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:16:19.568529277 +0000 UTC m=+1185.911577243" watchObservedRunningTime="2026-10-30 07:16:24.606096067 +0000 UTC m=+1190.949144033" Oct 30 07:16:25.556103 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:25.556063 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" event={"ID":"2d36f333-0885-434c-aea5-0f9f1d44217c","Type":"ContainerStarted","Data":"2575f1f581e517ee9f67147ec3cdb8a10f15329e58bedea791bbc20cdcbe57ed"} Oct 30 07:16:25.558064 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:25.558039 4951 generic.go:356] "Generic (PLEG): container finished" podID="8a282e3a-aa71-4b0a-b9db-f475645f94e6" containerID="993ee6f9224afbd09b035fb21c698416775eddccf8f94a896ef1e155d191c172" exitCode=0 Oct 30 07:16:25.558182 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:25.558114 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" event={"ID":"8a282e3a-aa71-4b0a-b9db-f475645f94e6","Type":"ContainerDied","Data":"993ee6f9224afbd09b035fb21c698416775eddccf8f94a896ef1e155d191c172"} Oct 30 07:16:25.558526 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:25.558510 4951 scope.go:122] "RemoveContainer" containerID="993ee6f9224afbd09b035fb21c698416775eddccf8f94a896ef1e155d191c172" Oct 30 07:16:25.560335 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:25.560299 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:16:25.560427 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:25.560409 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:16:25.560478 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:25.560462 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:16:25.802606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:25.801242 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd"] Oct 30 07:16:25.804791 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:25.804767 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd" Oct 30 07:16:25.862612 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:25.844913 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-1e3dd1-serving-cert\"" Oct 30 07:16:25.870940 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:25.870699 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd"] Oct 30 07:16:25.896595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:25.895146 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/369630cd-08d5-4b99-9a86-c33e774becf7-maas-api-tls\") pod \"maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd\" (UID: \"369630cd-08d5-4b99-9a86-c33e774becf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd" Oct 30 07:16:25.896595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:25.895270 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/369630cd-08d5-4b99-9a86-c33e774becf7-metrics-certs\") pod \"maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd\" (UID: \"369630cd-08d5-4b99-9a86-c33e774becf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd" Oct 30 07:16:25.896595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:25.895523 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/369630cd-08d5-4b99-9a86-c33e774becf7-openshift-service-ca\") pod \"maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd\" (UID: \"369630cd-08d5-4b99-9a86-c33e774becf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd" Oct 30 07:16:25.896595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:25.895635 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlbmj\" (UniqueName: \"kubernetes.io/projected/369630cd-08d5-4b99-9a86-c33e774becf7-kube-api-access-tlbmj\") pod \"maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd\" (UID: \"369630cd-08d5-4b99-9a86-c33e774becf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd" Oct 30 07:16:25.998872 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:25.997218 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/369630cd-08d5-4b99-9a86-c33e774becf7-openshift-service-ca\") pod \"maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd\" (UID: \"369630cd-08d5-4b99-9a86-c33e774becf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd" Oct 30 07:16:25.998872 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:25.997294 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlbmj\" (UniqueName: \"kubernetes.io/projected/369630cd-08d5-4b99-9a86-c33e774becf7-kube-api-access-tlbmj\") pod \"maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd\" (UID: \"369630cd-08d5-4b99-9a86-c33e774becf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd" Oct 30 07:16:25.998872 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:25.997352 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/369630cd-08d5-4b99-9a86-c33e774becf7-maas-api-tls\") pod \"maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd\" (UID: \"369630cd-08d5-4b99-9a86-c33e774becf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd" Oct 30 07:16:25.998872 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:25.997397 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/369630cd-08d5-4b99-9a86-c33e774becf7-metrics-certs\") pod \"maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd\" (UID: \"369630cd-08d5-4b99-9a86-c33e774becf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd" Oct 30 07:16:25.998872 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:25.998496 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/369630cd-08d5-4b99-9a86-c33e774becf7-openshift-service-ca\") pod \"maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd\" (UID: \"369630cd-08d5-4b99-9a86-c33e774becf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd" Oct 30 07:16:26.001017 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:26.000993 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/369630cd-08d5-4b99-9a86-c33e774becf7-maas-api-tls\") pod \"maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd\" (UID: \"369630cd-08d5-4b99-9a86-c33e774becf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd" Oct 30 07:16:26.001195 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:26.001171 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/369630cd-08d5-4b99-9a86-c33e774becf7-metrics-certs\") pod \"maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd\" (UID: \"369630cd-08d5-4b99-9a86-c33e774becf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd" Oct 30 07:16:26.012829 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:26.012798 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:16:26.013101 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:26.013076 4951 patch_prober.go:29] interesting pod/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 07:16:26.013274 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:26.013247 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" podUID="2d36f333-0885-434c-aea5-0f9f1d44217c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 07:16:26.020324 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:26.020290 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlbmj\" (UniqueName: \"kubernetes.io/projected/369630cd-08d5-4b99-9a86-c33e774becf7-kube-api-access-tlbmj\") pod \"maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd\" (UID: \"369630cd-08d5-4b99-9a86-c33e774becf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd" Oct 30 07:16:26.174055 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:26.172889 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd" Oct 30 07:16:26.629598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:26.615937 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" event={"ID":"8a282e3a-aa71-4b0a-b9db-f475645f94e6","Type":"ContainerStarted","Data":"05a08253454e72f3d45a4bf4e2e2297f5bc0d311cccf2f1865b4106dcd38704a"} Oct 30 07:16:26.768630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:26.757377 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:16:26.768630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:26.759774 4951 patch_prober.go:29] interesting pod/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 07:16:26.768630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:26.759835 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" podUID="8a282e3a-aa71-4b0a-b9db-f475645f94e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 07:16:26.804762 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:26.804492 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd"] Oct 30 07:16:27.019682 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:27.019562 4951 patch_prober.go:29] interesting pod/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 07:16:27.019862 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:27.019699 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" podUID="2d36f333-0885-434c-aea5-0f9f1d44217c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 07:16:27.650648 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:27.649610 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd" event={"ID":"369630cd-08d5-4b99-9a86-c33e774becf7","Type":"ContainerStarted","Data":"ea1fa2ba279379b5c03a5de510bfa905bbe16fa41005e98475058febce10cbb4"} Oct 30 07:16:27.650648 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:27.649669 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd" event={"ID":"369630cd-08d5-4b99-9a86-c33e774becf7","Type":"ContainerStarted","Data":"f02265ecc8e8816e92d49f60471aa94011bf7f188fb760ac2ed1add4f10c5406"} Oct 30 07:16:27.650648 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:27.649721 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd" Oct 30 07:16:27.747594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:27.741782 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd" podStartSLOduration=2.330050291 podStartE2EDuration="2.741759148s" totalImagesPullingTime="411.708857ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:16:27.69879707 +0000 UTC m=+1194.041845059" watchObservedRunningTime="2026-10-30 07:16:27.741759148 +0000 UTC m=+1194.084807114" Oct 30 07:16:27.755594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:27.753714 4951 patch_prober.go:29] interesting pod/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 07:16:27.755594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:27.753775 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" podUID="8a282e3a-aa71-4b0a-b9db-f475645f94e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 07:16:28.031465 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:28.031029 4951 patch_prober.go:29] interesting pod/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 07:16:28.031465 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:28.031091 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" podUID="2d36f333-0885-434c-aea5-0f9f1d44217c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 07:16:28.758594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:28.758008 4951 patch_prober.go:29] interesting pod/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 07:16:28.758594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:28.758071 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" podUID="8a282e3a-aa71-4b0a-b9db-f475645f94e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 07:16:29.013884 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:29.013789 4951 patch_prober.go:29] interesting pod/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 07:16:29.013884 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:29.013860 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" podUID="2d36f333-0885-434c-aea5-0f9f1d44217c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 07:16:29.758994 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:29.758955 4951 patch_prober.go:29] interesting pod/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 07:16:29.759606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:29.759553 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" podUID="8a282e3a-aa71-4b0a-b9db-f475645f94e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 07:16:30.013724 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:30.013648 4951 patch_prober.go:29] interesting pod/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 07:16:30.013920 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:30.013899 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" podUID="2d36f333-0885-434c-aea5-0f9f1d44217c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 07:16:30.505499 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:30.505467 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-1e3dd1-789c944fd4-m74hx" Oct 30 07:16:30.505707 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:30.505640 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-56eb7f-5fbf45d446-d6gzv" Oct 30 07:16:30.506703 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:30.506673 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e3dd1-695f9f69f9-w54sq" Oct 30 07:16:30.620815 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:30.620731 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs"] Oct 30 07:16:30.712188 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:30.704289 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs" Oct 30 07:16:30.712377 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:30.712310 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs"] Oct 30 07:16:30.768597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:30.767917 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-56eb7f-serving-cert\"" Oct 30 07:16:30.795561 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:30.793403 4951 patch_prober.go:29] interesting pod/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 07:16:30.795561 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:30.793505 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" podUID="8a282e3a-aa71-4b0a-b9db-f475645f94e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 07:16:30.801554 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:30.801526 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fabd32c8-62a7-44ac-9708-e5b56f695b12-metrics-certs\") pod \"maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs\" (UID: \"fabd32c8-62a7-44ac-9708-e5b56f695b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs" Oct 30 07:16:30.801706 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:30.801691 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fabd32c8-62a7-44ac-9708-e5b56f695b12-maas-api-tls\") pod \"maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs\" (UID: \"fabd32c8-62a7-44ac-9708-e5b56f695b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs" Oct 30 07:16:30.801749 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:30.801728 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wk6d\" (UniqueName: \"kubernetes.io/projected/fabd32c8-62a7-44ac-9708-e5b56f695b12-kube-api-access-5wk6d\") pod \"maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs\" (UID: \"fabd32c8-62a7-44ac-9708-e5b56f695b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs" Oct 30 07:16:30.801954 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:30.801928 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fabd32c8-62a7-44ac-9708-e5b56f695b12-openshift-service-ca\") pod \"maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs\" (UID: \"fabd32c8-62a7-44ac-9708-e5b56f695b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs" Oct 30 07:16:30.903841 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:30.903757 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wk6d\" (UniqueName: \"kubernetes.io/projected/fabd32c8-62a7-44ac-9708-e5b56f695b12-kube-api-access-5wk6d\") pod \"maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs\" (UID: \"fabd32c8-62a7-44ac-9708-e5b56f695b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs" Oct 30 07:16:30.904012 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:30.903896 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fabd32c8-62a7-44ac-9708-e5b56f695b12-openshift-service-ca\") pod \"maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs\" (UID: \"fabd32c8-62a7-44ac-9708-e5b56f695b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs" Oct 30 07:16:30.904012 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:30.903982 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fabd32c8-62a7-44ac-9708-e5b56f695b12-metrics-certs\") pod \"maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs\" (UID: \"fabd32c8-62a7-44ac-9708-e5b56f695b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs" Oct 30 07:16:30.904132 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:30.904117 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fabd32c8-62a7-44ac-9708-e5b56f695b12-maas-api-tls\") pod \"maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs\" (UID: \"fabd32c8-62a7-44ac-9708-e5b56f695b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs" Oct 30 07:16:30.904766 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:30.904734 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fabd32c8-62a7-44ac-9708-e5b56f695b12-openshift-service-ca\") pod \"maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs\" (UID: \"fabd32c8-62a7-44ac-9708-e5b56f695b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs" Oct 30 07:16:30.906608 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:30.906559 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fabd32c8-62a7-44ac-9708-e5b56f695b12-maas-api-tls\") pod \"maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs\" (UID: \"fabd32c8-62a7-44ac-9708-e5b56f695b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs" Oct 30 07:16:30.909182 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:30.909163 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fabd32c8-62a7-44ac-9708-e5b56f695b12-metrics-certs\") pod \"maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs\" (UID: \"fabd32c8-62a7-44ac-9708-e5b56f695b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs" Oct 30 07:16:30.976592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:30.976127 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wk6d\" (UniqueName: \"kubernetes.io/projected/fabd32c8-62a7-44ac-9708-e5b56f695b12-kube-api-access-5wk6d\") pod \"maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs\" (UID: \"fabd32c8-62a7-44ac-9708-e5b56f695b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs" Oct 30 07:16:31.017338 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.010651 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln"] Oct 30 07:16:31.018040 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.017987 4951 patch_prober.go:29] interesting pod/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 07:16:31.018254 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.018223 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" podUID="2d36f333-0885-434c-aea5-0f9f1d44217c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 07:16:31.071812 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.070432 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln"] Oct 30 07:16:31.072842 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.072819 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:16:31.127638 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.112162 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs" Oct 30 07:16:31.162729 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.129539 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-istio-data\") pod \"e2e-api-a-318524-openshift-default-779d745bc5-ftbln\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:16:31.162729 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.129639 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-workload-socket\") pod \"e2e-api-a-318524-openshift-default-779d745bc5-ftbln\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:16:31.162729 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.129726 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-credential-socket\") pod \"e2e-api-a-318524-openshift-default-779d745bc5-ftbln\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:16:31.162729 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.129808 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1471e3cc-b331-4f73-aef3-33af56f4fed8-istio-podinfo\") pod \"e2e-api-a-318524-openshift-default-779d745bc5-ftbln\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:16:31.162729 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.129997 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1471e3cc-b331-4f73-aef3-33af56f4fed8-istio-token\") pod \"e2e-api-a-318524-openshift-default-779d745bc5-ftbln\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:16:31.162729 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.130045 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlqlc\" (UniqueName: \"kubernetes.io/projected/1471e3cc-b331-4f73-aef3-33af56f4fed8-kube-api-access-nlqlc\") pod \"e2e-api-a-318524-openshift-default-779d745bc5-ftbln\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:16:31.162729 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.130082 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1471e3cc-b331-4f73-aef3-33af56f4fed8-istiod-ca-cert\") pod \"e2e-api-a-318524-openshift-default-779d745bc5-ftbln\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:16:31.162729 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.130156 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-istio-envoy\") pod \"e2e-api-a-318524-openshift-default-779d745bc5-ftbln\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:16:31.162729 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.130261 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-workload-certs\") pod \"e2e-api-a-318524-openshift-default-779d745bc5-ftbln\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:16:31.303626 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.298513 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlqlc\" (UniqueName: \"kubernetes.io/projected/1471e3cc-b331-4f73-aef3-33af56f4fed8-kube-api-access-nlqlc\") pod \"e2e-api-a-318524-openshift-default-779d745bc5-ftbln\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:16:31.303626 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.298589 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1471e3cc-b331-4f73-aef3-33af56f4fed8-istiod-ca-cert\") pod \"e2e-api-a-318524-openshift-default-779d745bc5-ftbln\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:16:31.303626 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.298651 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-istio-envoy\") pod \"e2e-api-a-318524-openshift-default-779d745bc5-ftbln\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:16:31.303626 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.298734 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-workload-certs\") pod \"e2e-api-a-318524-openshift-default-779d745bc5-ftbln\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:16:31.303626 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.298803 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-istio-data\") pod \"e2e-api-a-318524-openshift-default-779d745bc5-ftbln\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:16:31.303626 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.298855 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-workload-socket\") pod \"e2e-api-a-318524-openshift-default-779d745bc5-ftbln\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:16:31.303626 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.298922 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-credential-socket\") pod \"e2e-api-a-318524-openshift-default-779d745bc5-ftbln\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:16:31.303626 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.298950 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1471e3cc-b331-4f73-aef3-33af56f4fed8-istio-podinfo\") pod \"e2e-api-a-318524-openshift-default-779d745bc5-ftbln\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:16:31.303626 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.299083 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1471e3cc-b331-4f73-aef3-33af56f4fed8-istio-token\") pod \"e2e-api-a-318524-openshift-default-779d745bc5-ftbln\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:16:31.329916 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.315314 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1471e3cc-b331-4f73-aef3-33af56f4fed8-istiod-ca-cert\") pod \"e2e-api-a-318524-openshift-default-779d745bc5-ftbln\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:16:31.329916 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.319199 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-workload-socket\") pod \"e2e-api-a-318524-openshift-default-779d745bc5-ftbln\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:16:31.329916 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.319451 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-workload-certs\") pod \"e2e-api-a-318524-openshift-default-779d745bc5-ftbln\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:16:31.329916 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.319720 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-istio-data\") pod \"e2e-api-a-318524-openshift-default-779d745bc5-ftbln\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:16:31.329916 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.319986 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-credential-socket\") pod \"e2e-api-a-318524-openshift-default-779d745bc5-ftbln\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:16:31.372627 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.352455 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-318524-openshift-default-dockercfg-727xf\"" Oct 30 07:16:31.634955 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.634013 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1471e3cc-b331-4f73-aef3-33af56f4fed8-istio-token\") pod \"e2e-api-a-318524-openshift-default-779d745bc5-ftbln\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:16:31.647593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.639218 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1471e3cc-b331-4f73-aef3-33af56f4fed8-istio-podinfo\") pod \"e2e-api-a-318524-openshift-default-779d745bc5-ftbln\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:16:31.668701 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.668634 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-istio-envoy\") pod \"e2e-api-a-318524-openshift-default-779d745bc5-ftbln\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:16:31.702450 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.701159 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlqlc\" (UniqueName: \"kubernetes.io/projected/1471e3cc-b331-4f73-aef3-33af56f4fed8-kube-api-access-nlqlc\") pod \"e2e-api-a-318524-openshift-default-779d745bc5-ftbln\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:16:31.796720 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.786806 4951 patch_prober.go:29] interesting pod/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 07:16:31.796720 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.786875 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" podUID="8a282e3a-aa71-4b0a-b9db-f475645f94e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 07:16:31.926688 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:31.906330 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:16:32.172072 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:32.169923 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:16:32.172072 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:32.171850 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:16:32.200527 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:32.188955 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:16:32.563447 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:32.563414 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs"] Oct 30 07:16:32.745615 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:32.740633 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln"] Oct 30 07:16:32.764408 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:32.763300 4951 patch_prober.go:29] interesting pod/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 07:16:32.764408 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:32.763373 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" podUID="8a282e3a-aa71-4b0a-b9db-f475645f94e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 07:16:32.768663 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:32.768243 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:16:32.768663 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:32.768314 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:16:32.768663 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:32.768343 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:16:32.904774 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:32.904725 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs" event={"ID":"fabd32c8-62a7-44ac-9708-e5b56f695b12","Type":"ContainerStarted","Data":"1447116c78ed59e9d5d5e4e31672e83deab21b2a1da71c71a58f560d6b096abc"} Oct 30 07:16:32.907364 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:32.907331 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" event={"ID":"1471e3cc-b331-4f73-aef3-33af56f4fed8","Type":"ContainerStarted","Data":"04252430b67c2e0fcb118208575eb0e0566410813ce043a880bed4b65d6372ea"} Oct 30 07:16:32.907502 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:32.907417 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" event={"ID":"1471e3cc-b331-4f73-aef3-33af56f4fed8","Type":"ContainerStarted","Data":"c1ab3de76dc2b4e54b275f7e6f487dea121167252d6fff791d7fc321e2c7957e"} Oct 30 07:16:32.953143 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:32.953064 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" podStartSLOduration=2.9530431139999997 podStartE2EDuration="2.953043114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:16:32.950520284 +0000 UTC m=+1199.293568250" watchObservedRunningTime="2026-10-30 07:16:32.953043114 +0000 UTC m=+1199.296091081" Oct 30 07:16:32.958500 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:32.958470 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:16:33.685665 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:33.685637 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd" Oct 30 07:16:33.757286 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:33.757252 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:16:33.906984 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:33.906956 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:16:33.912909 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:33.912880 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:16:33.917782 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:33.917742 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs" event={"ID":"fabd32c8-62a7-44ac-9708-e5b56f695b12","Type":"ContainerStarted","Data":"da119fb51dbbb6eb9045c67ee941797015823e331413311c0a5d818d35a05c2f"} Oct 30 07:16:33.917948 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:33.917838 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs" Oct 30 07:16:33.918982 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:33.918921 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:16:33.919259 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:33.919237 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:16:33.919837 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:33.919818 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:16:34.089674 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:34.089610 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs" podStartSLOduration=3.779422345 podStartE2EDuration="4.089563425s" totalImagesPullingTime="310.14108ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:16:34.087122774 +0000 UTC m=+1200.430170739" watchObservedRunningTime="2026-10-30 07:16:34.089563425 +0000 UTC m=+1200.432611389" Oct 30 07:16:34.102005 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:34.101979 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/3.log" Oct 30 07:16:34.114515 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:34.114472 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 07:16:34.118962 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:34.118918 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/3.log" Oct 30 07:16:34.125056 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:34.125030 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 07:16:37.548413 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:37.548378 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-9bfvf"] Oct 30 07:16:39.932366 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:39.932336 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs" Oct 30 07:16:42.551690 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:42.551636 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-9bfvf" podUID="8bc553e5-f6d7-49e4-b4e1-d26f35e53cad" containerName="payload-processing" containerID="cri-o://d6d0a32cdbb5aa3e7092fd14f3a001aa894a11eb341c5be3a8f3a5c0c6019eb4" gracePeriod=25 Oct 30 07:16:43.218653 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:43.218195 4951 generic.go:356] "Generic (PLEG): container finished" podID="8bc553e5-f6d7-49e4-b4e1-d26f35e53cad" containerID="d6d0a32cdbb5aa3e7092fd14f3a001aa894a11eb341c5be3a8f3a5c0c6019eb4" exitCode=0 Oct 30 07:16:43.218653 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:43.218250 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-9bfvf" event={"ID":"8bc553e5-f6d7-49e4-b4e1-d26f35e53cad","Type":"ContainerDied","Data":"d6d0a32cdbb5aa3e7092fd14f3a001aa894a11eb341c5be3a8f3a5c0c6019eb4"} Oct 30 07:16:43.501177 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:43.501145 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-9bfvf" Oct 30 07:16:43.574416 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:43.574199 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bc553e5-f6d7-49e4-b4e1-d26f35e53cad-kube-api-access-ljzs4\" (UniqueName: \"kubernetes.io/projected/8bc553e5-f6d7-49e4-b4e1-d26f35e53cad-kube-api-access-ljzs4\") pod \"8bc553e5-f6d7-49e4-b4e1-d26f35e53cad\" (UID: \"8bc553e5-f6d7-49e4-b4e1-d26f35e53cad\") " Oct 30 07:16:43.574416 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:43.574313 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bc553e5-f6d7-49e4-b4e1-d26f35e53cad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bc553e5-f6d7-49e4-b4e1-d26f35e53cad-plugins-config-volume\") pod \"8bc553e5-f6d7-49e4-b4e1-d26f35e53cad\" (UID: \"8bc553e5-f6d7-49e4-b4e1-d26f35e53cad\") " Oct 30 07:16:43.619680 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:43.602894 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bc553e5-f6d7-49e4-b4e1-d26f35e53cad-plugins-config-volume" pod "8bc553e5-f6d7-49e4-b4e1-d26f35e53cad" (UID: "8bc553e5-f6d7-49e4-b4e1-d26f35e53cad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:16:43.619680 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:43.619207 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bc553e5-f6d7-49e4-b4e1-d26f35e53cad-kube-api-access-ljzs4" pod "8bc553e5-f6d7-49e4-b4e1-d26f35e53cad" (UID: "8bc553e5-f6d7-49e4-b4e1-d26f35e53cad"). InnerVolumeSpecName "kube-api-access-ljzs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:16:43.704092 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:43.676816 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljzs4\" (UniqueName: \"kubernetes.io/projected/8bc553e5-f6d7-49e4-b4e1-d26f35e53cad-kube-api-access-ljzs4\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:16:43.704092 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:43.676867 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bc553e5-f6d7-49e4-b4e1-d26f35e53cad-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:16:44.315514 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:44.314424 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-9bfvf" event={"ID":"8bc553e5-f6d7-49e4-b4e1-d26f35e53cad","Type":"ContainerDied","Data":"2277c7166a865e41f0ec759815c8f887ec29bf4d781bf4671fe6a403da3e0d1a"} Oct 30 07:16:44.315514 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:44.314501 4951 scope.go:122] "RemoveContainer" containerID="d6d0a32cdbb5aa3e7092fd14f3a001aa894a11eb341c5be3a8f3a5c0c6019eb4" Oct 30 07:16:44.315514 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:44.314688 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-9bfvf" Oct 30 07:16:44.383512 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:44.383473 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-9bfvf"] Oct 30 07:16:44.395691 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:44.395649 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-9bfvf"] Oct 30 07:16:45.785989 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:45.785917 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bc553e5-f6d7-49e4-b4e1-d26f35e53cad" path="/var/lib/kubelet/pods/8bc553e5-f6d7-49e4-b4e1-d26f35e53cad/volumes" Oct 30 07:16:51.627962 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:51.627927 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl"] Oct 30 07:16:51.628454 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:51.628196 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl" podUID="2b9ab9fc-ba02-4743-9105-0140368dd943" containerName="maas-api" containerID="cri-o://7aa33001bb4961850ac38f7c610f8ce5f3b6c148cca6fdaf40f7d1fc587301c2" gracePeriod=30 Oct 30 07:16:52.265936 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.265891 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl" Oct 30 07:16:52.350390 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.350337 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b9ab9fc-ba02-4743-9105-0140368dd943-kube-api-access-5n25t\" (UniqueName: \"kubernetes.io/projected/2b9ab9fc-ba02-4743-9105-0140368dd943-kube-api-access-5n25t\") pod \"2b9ab9fc-ba02-4743-9105-0140368dd943\" (UID: \"2b9ab9fc-ba02-4743-9105-0140368dd943\") " Oct 30 07:16:52.350641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.350406 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b9ab9fc-ba02-4743-9105-0140368dd943-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b9ab9fc-ba02-4743-9105-0140368dd943-metrics-certs\") pod \"2b9ab9fc-ba02-4743-9105-0140368dd943\" (UID: \"2b9ab9fc-ba02-4743-9105-0140368dd943\") " Oct 30 07:16:52.350641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.350471 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b9ab9fc-ba02-4743-9105-0140368dd943-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b9ab9fc-ba02-4743-9105-0140368dd943-openshift-service-ca\") pod \"2b9ab9fc-ba02-4743-9105-0140368dd943\" (UID: \"2b9ab9fc-ba02-4743-9105-0140368dd943\") " Oct 30 07:16:52.350805 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.350641 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b9ab9fc-ba02-4743-9105-0140368dd943-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b9ab9fc-ba02-4743-9105-0140368dd943-maas-api-tls\") pod \"2b9ab9fc-ba02-4743-9105-0140368dd943\" (UID: \"2b9ab9fc-ba02-4743-9105-0140368dd943\") " Oct 30 07:16:52.361658 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.356926 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b9ab9fc-ba02-4743-9105-0140368dd943-openshift-service-ca" pod "2b9ab9fc-ba02-4743-9105-0140368dd943" (UID: "2b9ab9fc-ba02-4743-9105-0140368dd943"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:16:52.444299 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.444147 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b9ab9fc-ba02-4743-9105-0140368dd943-metrics-certs" pod "2b9ab9fc-ba02-4743-9105-0140368dd943" (UID: "2b9ab9fc-ba02-4743-9105-0140368dd943"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:16:52.444861 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.444819 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b9ab9fc-ba02-4743-9105-0140368dd943-maas-api-tls" pod "2b9ab9fc-ba02-4743-9105-0140368dd943" (UID: "2b9ab9fc-ba02-4743-9105-0140368dd943"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:16:52.450038 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.449987 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b9ab9fc-ba02-4743-9105-0140368dd943-kube-api-access-5n25t" pod "2b9ab9fc-ba02-4743-9105-0140368dd943" (UID: "2b9ab9fc-ba02-4743-9105-0140368dd943"). InnerVolumeSpecName "kube-api-access-5n25t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:16:52.453385 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.453304 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n25t\" (UniqueName: \"kubernetes.io/projected/2b9ab9fc-ba02-4743-9105-0140368dd943-kube-api-access-5n25t\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:16:52.453385 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.453336 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b9ab9fc-ba02-4743-9105-0140368dd943-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:16:52.453385 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.453348 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b9ab9fc-ba02-4743-9105-0140368dd943-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:16:52.453385 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.453362 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b9ab9fc-ba02-4743-9105-0140368dd943-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:16:52.474617 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.474142 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-318524-65648b9ff6-7dz6g"] Oct 30 07:16:52.476399 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.475219 4951 cpu_manager.go:414] "containerMap: removing container" podUID="8bc553e5-f6d7-49e4-b4e1-d26f35e53cad" containerName="payload-processing" Oct 30 07:16:52.476399 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.475242 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bc553e5-f6d7-49e4-b4e1-d26f35e53cad" containerName="payload-processing" Oct 30 07:16:52.476399 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.475298 4951 cpu_manager.go:414] "containerMap: removing container" podUID="2b9ab9fc-ba02-4743-9105-0140368dd943" containerName="maas-api" Oct 30 07:16:52.476399 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.475306 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b9ab9fc-ba02-4743-9105-0140368dd943" containerName="maas-api" Oct 30 07:16:52.476399 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.475440 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bc553e5-f6d7-49e4-b4e1-d26f35e53cad" containerName="payload-processing" Oct 30 07:16:52.476399 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.475459 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b9ab9fc-ba02-4743-9105-0140368dd943" containerName="maas-api" Oct 30 07:16:52.539175 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.505939 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-318524-65648b9ff6-7dz6g" Oct 30 07:16:52.539175 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.507523 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-318524-65648b9ff6-7dz6g"] Oct 30 07:16:52.541148 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.540208 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-318524-serving-cert\"" Oct 30 07:16:52.561818 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.561320 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdad6c53-2e5e-4f0f-b5a5-598b27715e11-maas-api-tls\") pod \"maas-api-e2e-api-a-318524-65648b9ff6-7dz6g\" (UID: \"cdad6c53-2e5e-4f0f-b5a5-598b27715e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-318524-65648b9ff6-7dz6g" Oct 30 07:16:52.561818 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.561442 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdad6c53-2e5e-4f0f-b5a5-598b27715e11-metrics-certs\") pod \"maas-api-e2e-api-a-318524-65648b9ff6-7dz6g\" (UID: \"cdad6c53-2e5e-4f0f-b5a5-598b27715e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-318524-65648b9ff6-7dz6g" Oct 30 07:16:52.561818 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.561609 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht969\" (UniqueName: \"kubernetes.io/projected/cdad6c53-2e5e-4f0f-b5a5-598b27715e11-kube-api-access-ht969\") pod \"maas-api-e2e-api-a-318524-65648b9ff6-7dz6g\" (UID: \"cdad6c53-2e5e-4f0f-b5a5-598b27715e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-318524-65648b9ff6-7dz6g" Oct 30 07:16:52.561818 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.561781 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdad6c53-2e5e-4f0f-b5a5-598b27715e11-openshift-service-ca\") pod \"maas-api-e2e-api-a-318524-65648b9ff6-7dz6g\" (UID: \"cdad6c53-2e5e-4f0f-b5a5-598b27715e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-318524-65648b9ff6-7dz6g" Oct 30 07:16:52.598594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.596474 4951 generic.go:356] "Generic (PLEG): container finished" podID="2b9ab9fc-ba02-4743-9105-0140368dd943" containerID="7aa33001bb4961850ac38f7c610f8ce5f3b6c148cca6fdaf40f7d1fc587301c2" exitCode=0 Oct 30 07:16:52.598594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.596531 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl" event={"ID":"2b9ab9fc-ba02-4743-9105-0140368dd943","Type":"ContainerDied","Data":"7aa33001bb4961850ac38f7c610f8ce5f3b6c148cca6fdaf40f7d1fc587301c2"} Oct 30 07:16:52.598594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.596593 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl" event={"ID":"2b9ab9fc-ba02-4743-9105-0140368dd943","Type":"ContainerDied","Data":"eaced34667471c7b33824da6fa07046f900881f3c3bf9d3bfdb4da7849c6568a"} Oct 30 07:16:52.598594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.596622 4951 scope.go:122] "RemoveContainer" containerID="7aa33001bb4961850ac38f7c610f8ce5f3b6c148cca6fdaf40f7d1fc587301c2" Oct 30 07:16:52.598594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.596782 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl" Oct 30 07:16:52.620246 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.618274 4951 scope.go:122] "RemoveContainer" containerID="7aa33001bb4961850ac38f7c610f8ce5f3b6c148cca6fdaf40f7d1fc587301c2" Oct 30 07:16:52.620648 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:16:52.620611 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aa33001bb4961850ac38f7c610f8ce5f3b6c148cca6fdaf40f7d1fc587301c2\": container with ID starting with 7aa33001bb4961850ac38f7c610f8ce5f3b6c148cca6fdaf40f7d1fc587301c2 not found: ID does not exist" containerID="7aa33001bb4961850ac38f7c610f8ce5f3b6c148cca6fdaf40f7d1fc587301c2" Oct 30 07:16:52.620739 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.620659 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aa33001bb4961850ac38f7c610f8ce5f3b6c148cca6fdaf40f7d1fc587301c2"} err="failed to get container status \"7aa33001bb4961850ac38f7c610f8ce5f3b6c148cca6fdaf40f7d1fc587301c2\": rpc error: code = NotFound desc = could not find container \"7aa33001bb4961850ac38f7c610f8ce5f3b6c148cca6fdaf40f7d1fc587301c2\": container with ID starting with 7aa33001bb4961850ac38f7c610f8ce5f3b6c148cca6fdaf40f7d1fc587301c2 not found: ID does not exist" Oct 30 07:16:52.667739 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.667702 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdad6c53-2e5e-4f0f-b5a5-598b27715e11-maas-api-tls\") pod \"maas-api-e2e-api-a-318524-65648b9ff6-7dz6g\" (UID: \"cdad6c53-2e5e-4f0f-b5a5-598b27715e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-318524-65648b9ff6-7dz6g" Oct 30 07:16:52.668269 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.667829 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdad6c53-2e5e-4f0f-b5a5-598b27715e11-metrics-certs\") pod \"maas-api-e2e-api-a-318524-65648b9ff6-7dz6g\" (UID: \"cdad6c53-2e5e-4f0f-b5a5-598b27715e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-318524-65648b9ff6-7dz6g" Oct 30 07:16:52.668269 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.668026 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht969\" (UniqueName: \"kubernetes.io/projected/cdad6c53-2e5e-4f0f-b5a5-598b27715e11-kube-api-access-ht969\") pod \"maas-api-e2e-api-a-318524-65648b9ff6-7dz6g\" (UID: \"cdad6c53-2e5e-4f0f-b5a5-598b27715e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-318524-65648b9ff6-7dz6g" Oct 30 07:16:52.668269 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.668112 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdad6c53-2e5e-4f0f-b5a5-598b27715e11-openshift-service-ca\") pod \"maas-api-e2e-api-a-318524-65648b9ff6-7dz6g\" (UID: \"cdad6c53-2e5e-4f0f-b5a5-598b27715e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-318524-65648b9ff6-7dz6g" Oct 30 07:16:52.668833 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.668804 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl"] Oct 30 07:16:52.669098 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.669072 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdad6c53-2e5e-4f0f-b5a5-598b27715e11-openshift-service-ca\") pod \"maas-api-e2e-api-a-318524-65648b9ff6-7dz6g\" (UID: \"cdad6c53-2e5e-4f0f-b5a5-598b27715e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-318524-65648b9ff6-7dz6g" Oct 30 07:16:52.670850 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.670820 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdad6c53-2e5e-4f0f-b5a5-598b27715e11-metrics-certs\") pod \"maas-api-e2e-api-a-318524-65648b9ff6-7dz6g\" (UID: \"cdad6c53-2e5e-4f0f-b5a5-598b27715e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-318524-65648b9ff6-7dz6g" Oct 30 07:16:52.672039 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.672012 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdad6c53-2e5e-4f0f-b5a5-598b27715e11-maas-api-tls\") pod \"maas-api-e2e-api-a-318524-65648b9ff6-7dz6g\" (UID: \"cdad6c53-2e5e-4f0f-b5a5-598b27715e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-318524-65648b9ff6-7dz6g" Oct 30 07:16:52.674760 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.674732 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-jcsbl"] Oct 30 07:16:52.694876 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.694797 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht969\" (UniqueName: \"kubernetes.io/projected/cdad6c53-2e5e-4f0f-b5a5-598b27715e11-kube-api-access-ht969\") pod \"maas-api-e2e-api-a-318524-65648b9ff6-7dz6g\" (UID: \"cdad6c53-2e5e-4f0f-b5a5-598b27715e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-318524-65648b9ff6-7dz6g" Oct 30 07:16:52.930374 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:52.930336 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-318524-65648b9ff6-7dz6g" Oct 30 07:16:53.039065 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:16:53.039026 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:16:53.238841 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:53.238814 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-318524-65648b9ff6-7dz6g"] Oct 30 07:16:53.240742 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:16:53.240712 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdad6c53_2e5e_4f0f_b5a5_598b27715e11.slice/crio-464a83dff8c5527b51c28895f12bc7b4d6649febc324112748335e5dc07f4590 WatchSource:0}: Error finding container 464a83dff8c5527b51c28895f12bc7b4d6649febc324112748335e5dc07f4590: Status 404 returned error can't find the container with id 464a83dff8c5527b51c28895f12bc7b4d6649febc324112748335e5dc07f4590 Oct 30 07:16:53.803845 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:53.803819 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b9ab9fc-ba02-4743-9105-0140368dd943" path="/var/lib/kubelet/pods/2b9ab9fc-ba02-4743-9105-0140368dd943/volumes" Oct 30 07:16:53.804219 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:53.804200 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-318524-65648b9ff6-7dz6g" event={"ID":"cdad6c53-2e5e-4f0f-b5a5-598b27715e11","Type":"ContainerStarted","Data":"464a83dff8c5527b51c28895f12bc7b4d6649febc324112748335e5dc07f4590"} Oct 30 07:16:54.823176 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:54.823143 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-318524-65648b9ff6-7dz6g" event={"ID":"cdad6c53-2e5e-4f0f-b5a5-598b27715e11","Type":"ContainerStarted","Data":"a2ea7bb317399579db4a11e37d2b854f93e40f9b8411ca65cf43e7dedfcc3556"} Oct 30 07:16:54.823660 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:54.823274 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-318524-65648b9ff6-7dz6g" Oct 30 07:16:54.864990 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:54.864939 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-318524-65648b9ff6-7dz6g" podStartSLOduration=2.309989429 podStartE2EDuration="2.864922657s" totalImagesPullingTime="554.933228ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:16:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:16:54.862069641 +0000 UTC m=+1221.205117626" watchObservedRunningTime="2026-10-30 07:16:54.864922657 +0000 UTC m=+1221.207970663" Oct 30 07:16:55.282069 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.282032 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j"] Oct 30 07:16:55.289823 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.289795 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:16:55.306555 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.306494 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9"] Oct 30 07:16:55.313501 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.313475 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:16:55.341600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.337755 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-56eb7f-openshift-default-dockercfg-qcrln\"" Oct 30 07:16:55.342355 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.342283 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-credential-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:16:55.342599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.342547 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-istio-data\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:16:55.342830 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.342769 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-workload-certs\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:16:55.342987 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.342811 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-workload-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:16:55.342987 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.342948 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c193cba6-1789-4935-835b-80afc22f58ad-istiod-ca-cert\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:16:55.343298 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.343243 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c193cba6-1789-4935-835b-80afc22f58ad-istio-token\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:16:55.343453 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.343397 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55jsx\" (UniqueName: \"kubernetes.io/projected/c193cba6-1789-4935-835b-80afc22f58ad-kube-api-access-55jsx\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:16:55.343588 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.343549 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-istio-envoy\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:16:55.344005 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.343775 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c193cba6-1789-4935-835b-80afc22f58ad-istio-podinfo\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:16:55.351521 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.351496 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j"] Oct 30 07:16:55.369702 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.369656 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9"] Oct 30 07:16:55.445920 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.445880 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c193cba6-1789-4935-835b-80afc22f58ad-istio-podinfo\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:16:55.446130 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.445956 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-istio-envoy\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:16:55.446130 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.445987 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67817544-1eb4-4e44-ba6c-b84c63df5f76-istiod-ca-cert\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:16:55.446130 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.446086 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67817544-1eb4-4e44-ba6c-b84c63df5f76-istio-token\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:16:55.446130 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.446120 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-workload-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:16:55.446312 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.446208 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-credential-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:16:55.446312 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.446259 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-istio-data\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:16:55.446417 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.446359 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-istio-data\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:16:55.446417 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.446400 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdwzx\" (UniqueName: \"kubernetes.io/projected/67817544-1eb4-4e44-ba6c-b84c63df5f76-kube-api-access-mdwzx\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:16:55.446528 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.446442 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-workload-certs\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:16:55.446528 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.446468 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-workload-certs\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:16:55.446528 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.446494 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-workload-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:16:55.446721 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.446693 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c193cba6-1789-4935-835b-80afc22f58ad-istiod-ca-cert\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:16:55.446800 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.446781 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67817544-1eb4-4e44-ba6c-b84c63df5f76-istio-podinfo\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:16:55.446861 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.446847 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-credential-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:16:55.446920 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.446905 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c193cba6-1789-4935-835b-80afc22f58ad-istio-token\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:16:55.446986 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.446971 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55jsx\" (UniqueName: \"kubernetes.io/projected/c193cba6-1789-4935-835b-80afc22f58ad-kube-api-access-55jsx\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:16:55.447047 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.447031 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-istio-envoy\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:16:55.450647 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.449945 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-istio-envoy\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:16:55.450647 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.449994 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-credential-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:16:55.450647 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.450020 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-istio-data\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:16:55.450647 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.450073 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-workload-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:16:55.450647 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.450095 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-workload-certs\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:16:55.450647 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.450346 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c193cba6-1789-4935-835b-80afc22f58ad-istio-podinfo\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:16:55.450647 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.450376 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c193cba6-1789-4935-835b-80afc22f58ad-istiod-ca-cert\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:16:55.470264 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.470226 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c193cba6-1789-4935-835b-80afc22f58ad-istio-token\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:16:55.470453 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.470239 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55jsx\" (UniqueName: \"kubernetes.io/projected/c193cba6-1789-4935-835b-80afc22f58ad-kube-api-access-55jsx\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:16:55.548657 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.548553 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-istio-envoy\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:16:55.548657 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.548604 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67817544-1eb4-4e44-ba6c-b84c63df5f76-istiod-ca-cert\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:16:55.548657 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.548650 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67817544-1eb4-4e44-ba6c-b84c63df5f76-istio-token\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:16:55.548980 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.548669 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-workload-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:16:55.548980 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.548717 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-istio-data\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:16:55.548980 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.548765 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdwzx\" (UniqueName: \"kubernetes.io/projected/67817544-1eb4-4e44-ba6c-b84c63df5f76-kube-api-access-mdwzx\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:16:55.548980 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.548823 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-workload-certs\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:16:55.548980 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.548898 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67817544-1eb4-4e44-ba6c-b84c63df5f76-istio-podinfo\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:16:55.548980 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.548956 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-credential-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:16:55.549279 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.549127 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-workload-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:16:55.549420 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.549401 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67817544-1eb4-4e44-ba6c-b84c63df5f76-istiod-ca-cert\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:16:55.549477 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.549394 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-istio-data\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:16:55.549530 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.549477 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-credential-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:16:55.549619 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.549603 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-workload-certs\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:16:55.551065 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.551037 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-istio-envoy\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:16:55.551388 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.551372 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67817544-1eb4-4e44-ba6c-b84c63df5f76-istio-podinfo\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:16:55.571407 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.571365 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67817544-1eb4-4e44-ba6c-b84c63df5f76-istio-token\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:16:55.574556 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.574524 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdwzx\" (UniqueName: \"kubernetes.io/projected/67817544-1eb4-4e44-ba6c-b84c63df5f76-kube-api-access-mdwzx\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:16:55.661884 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.661845 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:16:55.677856 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:55.677772 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:16:56.027008 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:56.020253 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:16:56.027008 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:56.020320 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:16:56.027008 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:56.020350 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:16:56.027008 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:56.021716 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j"] Oct 30 07:16:56.113735 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:56.113708 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9"] Oct 30 07:16:56.115463 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:56.115426 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:16:56.115602 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:56.115516 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:16:56.115602 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:56.115563 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:16:56.881596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:56.880615 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" event={"ID":"c193cba6-1789-4935-835b-80afc22f58ad","Type":"ContainerStarted","Data":"f456dd4a2e31710d2c1418b6b27d835ebb38edcb7d51c8e26c04fbdae0af7f5c"} Oct 30 07:16:56.881596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:56.880689 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" event={"ID":"c193cba6-1789-4935-835b-80afc22f58ad","Type":"ContainerStarted","Data":"01387206b1ce8a89a81239f5072276167d11ab18f3930e360e7f2ac41737cf42"} Oct 30 07:16:56.883603 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:56.883200 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" event={"ID":"67817544-1eb4-4e44-ba6c-b84c63df5f76","Type":"ContainerStarted","Data":"d13f424ae56f9715ca3ff73c2ed395644125fa23e08d9a093dfaf9bf0971b704"} Oct 30 07:16:56.883603 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:56.883262 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" event={"ID":"67817544-1eb4-4e44-ba6c-b84c63df5f76","Type":"ContainerStarted","Data":"43c9af56b50449a557a67c063aad5638c6fb82fdb60d3e3be834a28bc5845c2c"} Oct 30 07:16:56.960596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:56.959670 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" podStartSLOduration=1.959645498 podStartE2EDuration="1.959645498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:16:56.945320421 +0000 UTC m=+1223.288368452" watchObservedRunningTime="2026-10-30 07:16:56.959645498 +0000 UTC m=+1223.302693465" Oct 30 07:16:57.061597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:57.060747 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" podStartSLOduration=2.060724622 podStartE2EDuration="2.060724622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:16:57.059195362 +0000 UTC m=+1223.402243353" watchObservedRunningTime="2026-10-30 07:16:57.060724622 +0000 UTC m=+1223.403772589" Oct 30 07:16:57.669677 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:57.669638 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:16:57.679322 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:57.673720 4951 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 07:16:57.679322 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:57.673795 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" podUID="c193cba6-1789-4935-835b-80afc22f58ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 07:16:57.684169 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:57.684133 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:16:57.684556 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:57.684534 4951 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 07:16:57.684779 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:57.684756 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" podUID="67817544-1eb4-4e44-ba6c-b84c63df5f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 07:16:58.674597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:58.674049 4951 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 07:16:58.674597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:58.674116 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" podUID="c193cba6-1789-4935-835b-80afc22f58ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 07:16:58.680003 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:58.679965 4951 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 07:16:58.680260 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:58.680235 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" podUID="67817544-1eb4-4e44-ba6c-b84c63df5f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 07:16:59.672729 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:59.672690 4951 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 07:16:59.673029 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:59.672997 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" podUID="c193cba6-1789-4935-835b-80afc22f58ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 07:16:59.678942 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:59.678915 4951 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 07:16:59.679442 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:16:59.679417 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" podUID="67817544-1eb4-4e44-ba6c-b84c63df5f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 07:17:00.666014 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:00.665969 4951 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 07:17:00.666314 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:00.666282 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" podUID="c193cba6-1789-4935-835b-80afc22f58ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 07:17:00.679954 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:00.679921 4951 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 07:17:00.680475 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:00.680446 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" podUID="67817544-1eb4-4e44-ba6c-b84c63df5f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 07:17:00.872598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:00.871768 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-318524-65648b9ff6-7dz6g" Oct 30 07:17:01.665087 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:01.665047 4951 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 07:17:01.665377 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:01.665346 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" podUID="c193cba6-1789-4935-835b-80afc22f58ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 07:17:01.685516 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:01.685483 4951 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 07:17:01.686140 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:01.686110 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" podUID="67817544-1eb4-4e44-ba6c-b84c63df5f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 07:17:02.670299 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:02.668932 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:17:02.670643 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:02.670612 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:17:02.671965 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:02.671912 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:17:02.683410 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:02.683379 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:17:02.683692 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:02.683671 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:17:02.684370 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:02.684347 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:17:04.414711 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:04.414665 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97"] Oct 30 07:17:04.473889 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:04.473850 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97"] Oct 30 07:17:04.474074 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:04.474005 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:04.488448 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:04.488412 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cba7319-64a7-453f-95c7-ccf0c18033a0-istio-podinfo\") pod \"e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:04.488683 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:04.488509 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cba7319-64a7-453f-95c7-ccf0c18033a0-istiod-ca-cert\") pod \"e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:04.488683 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:04.488559 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-workload-certs\") pod \"e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:04.488683 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:04.488598 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-istio-envoy\") pod \"e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:04.488902 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:04.488722 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-workload-socket\") pod \"e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:04.488902 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:04.488823 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cba7319-64a7-453f-95c7-ccf0c18033a0-istio-token\") pod \"e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:04.489005 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:04.488900 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gzfs\" (UniqueName: \"kubernetes.io/projected/6cba7319-64a7-453f-95c7-ccf0c18033a0-kube-api-access-7gzfs\") pod \"e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:04.489005 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:04.488933 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-istio-data\") pod \"e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:04.489116 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:04.489023 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-credential-socket\") pod \"e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:04.490903 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:04.490877 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-6cc5f2-openshift-default-dockercfg-p26g2\"" Oct 30 07:17:04.598979 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:04.598937 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cba7319-64a7-453f-95c7-ccf0c18033a0-istiod-ca-cert\") pod \"e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:04.599264 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:04.599244 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-workload-certs\") pod \"e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:04.599382 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:04.599366 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-istio-envoy\") pod \"e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:04.603511 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:04.603477 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cba7319-64a7-453f-95c7-ccf0c18033a0-istiod-ca-cert\") pod \"e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:04.604917 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:04.604887 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-workload-certs\") pod \"e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:04.617043 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:04.611646 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-workload-socket\") pod \"e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:04.617542 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:04.617507 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cba7319-64a7-453f-95c7-ccf0c18033a0-istio-token\") pod \"e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:04.632746 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:04.632709 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gzfs\" (UniqueName: \"kubernetes.io/projected/6cba7319-64a7-453f-95c7-ccf0c18033a0-kube-api-access-7gzfs\") pod \"e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:04.648708 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:04.648671 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-istio-data\") pod \"e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:04.649561 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:04.649529 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-credential-socket\") pod \"e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:04.675044 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:04.674914 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cba7319-64a7-453f-95c7-ccf0c18033a0-istio-podinfo\") pod \"e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:04.795622 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:04.611964 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-workload-socket\") pod \"e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:04.795622 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:04.673660 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-credential-socket\") pod \"e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:04.795622 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:04.649243 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-istio-data\") pod \"e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:04.840611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:04.824860 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gzfs\" (UniqueName: \"kubernetes.io/projected/6cba7319-64a7-453f-95c7-ccf0c18033a0-kube-api-access-7gzfs\") pod \"e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:04.841655 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:04.841565 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-istio-envoy\") pod \"e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:04.842032 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:04.842004 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cba7319-64a7-453f-95c7-ccf0c18033a0-istio-token\") pod \"e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:04.860592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:04.850682 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cba7319-64a7-453f-95c7-ccf0c18033a0-istio-podinfo\") pod \"e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:05.096258 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:05.096211 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:05.414981 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:17:05.414950 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cba7319_64a7_453f_95c7_ccf0c18033a0.slice/crio-3ab768ce7306223007af6009c8630d7c338e4e29e4d987c0200f6dcb8fb10aca WatchSource:0}: Error finding container 3ab768ce7306223007af6009c8630d7c338e4e29e4d987c0200f6dcb8fb10aca: Status 404 returned error can't find the container with id 3ab768ce7306223007af6009c8630d7c338e4e29e4d987c0200f6dcb8fb10aca Oct 30 07:17:05.415588 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:05.415553 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97"] Oct 30 07:17:05.417819 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:05.417788 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:17:05.417884 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:05.417850 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:17:05.417884 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:05.417880 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:17:06.107606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:06.107543 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" event={"ID":"6cba7319-64a7-453f-95c7-ccf0c18033a0","Type":"ContainerStarted","Data":"15a6be79554d14c24357a6dc1e551cb55affdfada740b82fee4d6b1982bdb138"} Oct 30 07:17:06.109300 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:06.109263 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" event={"ID":"6cba7319-64a7-453f-95c7-ccf0c18033a0","Type":"ContainerStarted","Data":"3ab768ce7306223007af6009c8630d7c338e4e29e4d987c0200f6dcb8fb10aca"} Oct 30 07:17:06.109610 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:06.109588 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:06.136928 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:06.136892 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:06.190190 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:06.184285 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" podStartSLOduration=2.184263478 podStartE2EDuration="2.184263478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:17:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:17:06.182922458 +0000 UTC m=+1232.525970456" watchObservedRunningTime="2026-10-30 07:17:06.184263478 +0000 UTC m=+1232.527311444" Oct 30 07:17:06.638068 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:06.638035 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7c51aa-784f94685d-469jj"] Oct 30 07:17:06.663894 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:06.663863 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7c51aa-5dd58b9fcf-q65zk"] Oct 30 07:17:07.124668 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:07.124604 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:07.125944 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:07.125916 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:11.665384 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:11.665304 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-7c51aa-784f94685d-469jj" podUID="5caded86-2432-42eb-8179-f88d95e73857" containerName="payload-processing" containerID="cri-o://0b1d7ab2ad79b1df04161a75428f0f3b0766ff67f8adaadced8817111b357c57" gracePeriod=25 Oct 30 07:17:11.666986 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:11.666838 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c51aa-5dd58b9fcf-q65zk" podUID="10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4" containerName="payload-pre-processing" containerID="cri-o://f57180a9c3a256d37db525447c1dfc9b4a3991955d851d28329acc93506de23b" gracePeriod=25 Oct 30 07:17:12.056596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:12.055732 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7c51aa-784f94685d-469jj" Oct 30 07:17:12.217660 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:12.204346 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5caded86-2432-42eb-8179-f88d95e73857-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5caded86-2432-42eb-8179-f88d95e73857-plugins-config-volume\") pod \"5caded86-2432-42eb-8179-f88d95e73857\" (UID: \"5caded86-2432-42eb-8179-f88d95e73857\") " Oct 30 07:17:12.217660 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:12.204623 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5caded86-2432-42eb-8179-f88d95e73857-kube-api-access-jfhf6\" (UniqueName: \"kubernetes.io/projected/5caded86-2432-42eb-8179-f88d95e73857-kube-api-access-jfhf6\") pod \"5caded86-2432-42eb-8179-f88d95e73857\" (UID: \"5caded86-2432-42eb-8179-f88d95e73857\") " Oct 30 07:17:12.245096 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:12.242954 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5caded86-2432-42eb-8179-f88d95e73857-plugins-config-volume" pod "5caded86-2432-42eb-8179-f88d95e73857" (UID: "5caded86-2432-42eb-8179-f88d95e73857"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:17:12.272624 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:12.270805 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5caded86-2432-42eb-8179-f88d95e73857-kube-api-access-jfhf6" pod "5caded86-2432-42eb-8179-f88d95e73857" (UID: "5caded86-2432-42eb-8179-f88d95e73857"). InnerVolumeSpecName "kube-api-access-jfhf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:17:12.290075 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:12.290018 4951 generic.go:356] "Generic (PLEG): container finished" podID="5caded86-2432-42eb-8179-f88d95e73857" containerID="0b1d7ab2ad79b1df04161a75428f0f3b0766ff67f8adaadced8817111b357c57" exitCode=0 Oct 30 07:17:12.290279 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:12.290230 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7c51aa-784f94685d-469jj" Oct 30 07:17:12.290483 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:12.290445 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7c51aa-784f94685d-469jj" event={"ID":"5caded86-2432-42eb-8179-f88d95e73857","Type":"ContainerDied","Data":"0b1d7ab2ad79b1df04161a75428f0f3b0766ff67f8adaadced8817111b357c57"} Oct 30 07:17:12.290688 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:12.290665 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7c51aa-784f94685d-469jj" event={"ID":"5caded86-2432-42eb-8179-f88d95e73857","Type":"ContainerDied","Data":"80d0b0c8bdc82850981e6f163cda13d8d0452c06ad2008052e7cbf22dcf0acd6"} Oct 30 07:17:12.290873 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:12.290859 4951 scope.go:122] "RemoveContainer" containerID="0b1d7ab2ad79b1df04161a75428f0f3b0766ff67f8adaadced8817111b357c57" Oct 30 07:17:12.294397 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:12.294371 4951 generic.go:356] "Generic (PLEG): container finished" podID="10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4" containerID="f57180a9c3a256d37db525447c1dfc9b4a3991955d851d28329acc93506de23b" exitCode=0 Oct 30 07:17:12.295005 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:12.294979 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c51aa-5dd58b9fcf-q65zk" event={"ID":"10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4","Type":"ContainerDied","Data":"f57180a9c3a256d37db525447c1dfc9b4a3991955d851d28329acc93506de23b"} Oct 30 07:17:12.308262 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:12.308247 4951 scope.go:122] "RemoveContainer" containerID="0b1d7ab2ad79b1df04161a75428f0f3b0766ff67f8adaadced8817111b357c57" Oct 30 07:17:12.308667 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:17:12.308644 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b1d7ab2ad79b1df04161a75428f0f3b0766ff67f8adaadced8817111b357c57\": container with ID starting with 0b1d7ab2ad79b1df04161a75428f0f3b0766ff67f8adaadced8817111b357c57 not found: ID does not exist" containerID="0b1d7ab2ad79b1df04161a75428f0f3b0766ff67f8adaadced8817111b357c57" Oct 30 07:17:12.308735 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:12.308683 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b1d7ab2ad79b1df04161a75428f0f3b0766ff67f8adaadced8817111b357c57"} err="failed to get container status \"0b1d7ab2ad79b1df04161a75428f0f3b0766ff67f8adaadced8817111b357c57\": rpc error: code = NotFound desc = could not find container \"0b1d7ab2ad79b1df04161a75428f0f3b0766ff67f8adaadced8817111b357c57\": container with ID starting with 0b1d7ab2ad79b1df04161a75428f0f3b0766ff67f8adaadced8817111b357c57 not found: ID does not exist" Oct 30 07:17:12.338788 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:12.338758 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c51aa-5dd58b9fcf-q65zk" Oct 30 07:17:12.355604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:12.344904 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5caded86-2432-42eb-8179-f88d95e73857-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:17:12.355604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:12.344958 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfhf6\" (UniqueName: \"kubernetes.io/projected/5caded86-2432-42eb-8179-f88d95e73857-kube-api-access-jfhf6\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:17:12.364683 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:12.364640 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7c51aa-784f94685d-469jj"] Oct 30 07:17:12.370300 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:12.370263 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7c51aa-784f94685d-469jj"] Oct 30 07:17:12.451594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:12.449086 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4-kube-api-access-zgv6b\" (UniqueName: \"kubernetes.io/projected/10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4-kube-api-access-zgv6b\") pod \"10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4\" (UID: \"10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4\") " Oct 30 07:17:12.451594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:12.449436 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4-plugins-config-volume\") pod \"10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4\" (UID: \"10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4\") " Oct 30 07:17:12.453116 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:12.453078 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4-plugins-config-volume" pod "10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4" (UID: "10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:17:12.455332 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:12.455306 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:17:12.458564 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:12.458546 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4-kube-api-access-zgv6b" pod "10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4" (UID: "10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4"). InnerVolumeSpecName "kube-api-access-zgv6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:17:12.558081 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:12.558049 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgv6b\" (UniqueName: \"kubernetes.io/projected/10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4-kube-api-access-zgv6b\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:17:13.365597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:13.363170 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c51aa-5dd58b9fcf-q65zk" event={"ID":"10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4","Type":"ContainerDied","Data":"c0d575676091d5ffabb1ea9c83bbea18481834578a767216ec33d6f882d9684d"} Oct 30 07:17:13.365597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:13.363247 4951 scope.go:122] "RemoveContainer" containerID="f57180a9c3a256d37db525447c1dfc9b4a3991955d851d28329acc93506de23b" Oct 30 07:17:13.365597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:13.363405 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c51aa-5dd58b9fcf-q65zk" Oct 30 07:17:13.478413 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:13.478376 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7c51aa-5dd58b9fcf-q65zk"] Oct 30 07:17:13.489406 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:13.489370 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7c51aa-5dd58b9fcf-q65zk"] Oct 30 07:17:13.773509 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:13.773480 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4" path="/var/lib/kubelet/pods/10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4/volumes" Oct 30 07:17:13.773948 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:13.773932 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5caded86-2432-42eb-8179-f88d95e73857" path="/var/lib/kubelet/pods/5caded86-2432-42eb-8179-f88d95e73857/volumes" Oct 30 07:17:19.961745 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:19.961688 4951 patch_prober.go:29] interesting pod/e2e-api-a-318524-openshift-default-779d745bc5-ftbln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:17:19.962200 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:19.961763 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" podUID="1471e3cc-b331-4f73-aef3-33af56f4fed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:17:20.471808 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:20.471777 4951 generic.go:356] "Generic (PLEG): container finished" podID="1471e3cc-b331-4f73-aef3-33af56f4fed8" containerID="04252430b67c2e0fcb118208575eb0e0566410813ce043a880bed4b65d6372ea" exitCode=0 Oct 30 07:17:20.472027 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:20.471848 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" event={"ID":"1471e3cc-b331-4f73-aef3-33af56f4fed8","Type":"ContainerDied","Data":"04252430b67c2e0fcb118208575eb0e0566410813ce043a880bed4b65d6372ea"} Oct 30 07:17:20.472396 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:20.472374 4951 scope.go:122] "RemoveContainer" containerID="04252430b67c2e0fcb118208575eb0e0566410813ce043a880bed4b65d6372ea" Oct 30 07:17:20.474207 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:20.474173 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:17:20.474297 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:20.474250 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:17:20.474341 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:20.474303 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:17:21.597594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:21.597183 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" event={"ID":"1471e3cc-b331-4f73-aef3-33af56f4fed8","Type":"ContainerStarted","Data":"62e6f0675b9381df95248814c7e83e06ad2a7b1e9692e256faf9fc9cabfb1523"} Oct 30 07:17:21.907456 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:21.907374 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:17:21.907641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:21.907536 4951 patch_prober.go:29] interesting pod/e2e-api-a-318524-openshift-default-779d745bc5-ftbln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 07:17:21.907641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:21.907612 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" podUID="1471e3cc-b331-4f73-aef3-33af56f4fed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 07:17:22.907697 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:22.907658 4951 patch_prober.go:29] interesting pod/e2e-api-a-318524-openshift-default-779d745bc5-ftbln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 07:17:22.908116 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:22.907725 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" podUID="1471e3cc-b331-4f73-aef3-33af56f4fed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 07:17:22.918564 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:22.918521 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp"] Oct 30 07:17:22.919787 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:22.919750 4951 cpu_manager.go:414] "containerMap: removing container" podUID="10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4" containerName="payload-pre-processing" Oct 30 07:17:22.919787 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:22.919778 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4" containerName="payload-pre-processing" Oct 30 07:17:22.920006 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:22.919862 4951 cpu_manager.go:414] "containerMap: removing container" podUID="5caded86-2432-42eb-8179-f88d95e73857" containerName="payload-processing" Oct 30 07:17:22.920006 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:22.919870 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5caded86-2432-42eb-8179-f88d95e73857" containerName="payload-processing" Oct 30 07:17:22.920107 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:22.920036 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="5caded86-2432-42eb-8179-f88d95e73857" containerName="payload-processing" Oct 30 07:17:22.920107 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:22.920053 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="10a59bb2-4606-4c0a-baf0-0d5fbabc1ba4" containerName="payload-pre-processing" Oct 30 07:17:22.966600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:22.964956 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp" Oct 30 07:17:22.994930 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:22.994890 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp"] Oct 30 07:17:23.068006 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:23.067969 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-6cc5f2-serving-cert\"" Oct 30 07:17:23.075114 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:17:23.071971 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:17:23.084614 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:23.084342 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8b8448b-c2f2-4e27-bea9-a5d4cd44466f-openshift-service-ca\") pod \"maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp\" (UID: \"a8b8448b-c2f2-4e27-bea9-a5d4cd44466f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp" Oct 30 07:17:23.084614 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:23.084508 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8b8448b-c2f2-4e27-bea9-a5d4cd44466f-maas-api-tls\") pod \"maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp\" (UID: \"a8b8448b-c2f2-4e27-bea9-a5d4cd44466f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp" Oct 30 07:17:23.084614 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:23.084549 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbzm5\" (UniqueName: \"kubernetes.io/projected/a8b8448b-c2f2-4e27-bea9-a5d4cd44466f-kube-api-access-vbzm5\") pod \"maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp\" (UID: \"a8b8448b-c2f2-4e27-bea9-a5d4cd44466f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp" Oct 30 07:17:23.084924 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:23.084767 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8b8448b-c2f2-4e27-bea9-a5d4cd44466f-metrics-certs\") pod \"maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp\" (UID: \"a8b8448b-c2f2-4e27-bea9-a5d4cd44466f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp" Oct 30 07:17:23.256666 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:23.250271 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8b8448b-c2f2-4e27-bea9-a5d4cd44466f-metrics-certs\") pod \"maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp\" (UID: \"a8b8448b-c2f2-4e27-bea9-a5d4cd44466f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp" Oct 30 07:17:23.256666 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:23.250485 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8b8448b-c2f2-4e27-bea9-a5d4cd44466f-openshift-service-ca\") pod \"maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp\" (UID: \"a8b8448b-c2f2-4e27-bea9-a5d4cd44466f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp" Oct 30 07:17:23.256666 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:23.250665 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8b8448b-c2f2-4e27-bea9-a5d4cd44466f-maas-api-tls\") pod \"maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp\" (UID: \"a8b8448b-c2f2-4e27-bea9-a5d4cd44466f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp" Oct 30 07:17:23.256666 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:23.250705 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbzm5\" (UniqueName: \"kubernetes.io/projected/a8b8448b-c2f2-4e27-bea9-a5d4cd44466f-kube-api-access-vbzm5\") pod \"maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp\" (UID: \"a8b8448b-c2f2-4e27-bea9-a5d4cd44466f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp" Oct 30 07:17:23.334598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:23.273664 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8b8448b-c2f2-4e27-bea9-a5d4cd44466f-openshift-service-ca\") pod \"maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp\" (UID: \"a8b8448b-c2f2-4e27-bea9-a5d4cd44466f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp" Oct 30 07:17:23.439203 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:23.437385 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbzm5\" (UniqueName: \"kubernetes.io/projected/a8b8448b-c2f2-4e27-bea9-a5d4cd44466f-kube-api-access-vbzm5\") pod \"maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp\" (UID: \"a8b8448b-c2f2-4e27-bea9-a5d4cd44466f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp" Oct 30 07:17:23.483598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:23.477474 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8b8448b-c2f2-4e27-bea9-a5d4cd44466f-maas-api-tls\") pod \"maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp\" (UID: \"a8b8448b-c2f2-4e27-bea9-a5d4cd44466f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp" Oct 30 07:17:23.513647 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:23.509883 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8b8448b-c2f2-4e27-bea9-a5d4cd44466f-metrics-certs\") pod \"maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp\" (UID: \"a8b8448b-c2f2-4e27-bea9-a5d4cd44466f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp" Oct 30 07:17:23.599309 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:23.599272 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp" Oct 30 07:17:23.908584 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:23.908475 4951 patch_prober.go:29] interesting pod/e2e-api-a-318524-openshift-default-779d745bc5-ftbln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 07:17:23.909167 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:23.909132 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" podUID="1471e3cc-b331-4f73-aef3-33af56f4fed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 07:17:24.346686 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:24.307596 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp"] Oct 30 07:17:24.732167 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:24.732063 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp" event={"ID":"a8b8448b-c2f2-4e27-bea9-a5d4cd44466f","Type":"ContainerStarted","Data":"f0ad5e9e56761c075dbf8bcb80c5d73234b1ac2f5dcecb2d6852bc258c92cdc2"} Oct 30 07:17:24.957631 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:24.955994 4951 patch_prober.go:29] interesting pod/e2e-api-a-318524-openshift-default-779d745bc5-ftbln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 07:17:24.957631 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:24.956058 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" podUID="1471e3cc-b331-4f73-aef3-33af56f4fed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 07:17:25.752378 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:25.752336 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp" event={"ID":"a8b8448b-c2f2-4e27-bea9-a5d4cd44466f","Type":"ContainerStarted","Data":"7ccfc7fa0d5e661db5a9921c332bc3688a43e9637f756eef37b4a71e1b7f8037"} Oct 30 07:17:25.752616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:25.752446 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp" Oct 30 07:17:25.795415 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:25.795362 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp" podStartSLOduration=3.329454042 podStartE2EDuration="3.79534491s" totalImagesPullingTime="465.890868ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:17:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:17:25.793986589 +0000 UTC m=+1252.137034571" watchObservedRunningTime="2026-10-30 07:17:25.79534491 +0000 UTC m=+1252.138392916" Oct 30 07:17:25.913597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:25.912521 4951 patch_prober.go:29] interesting pod/e2e-api-a-318524-openshift-default-779d745bc5-ftbln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 07:17:25.913597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:25.912609 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" podUID="1471e3cc-b331-4f73-aef3-33af56f4fed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 07:17:26.908150 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:26.908105 4951 patch_prober.go:29] interesting pod/e2e-api-a-318524-openshift-default-779d745bc5-ftbln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 07:17:26.908557 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:26.908176 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" podUID="1471e3cc-b331-4f73-aef3-33af56f4fed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 07:17:27.911315 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:27.911281 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:17:27.911817 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:27.911692 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:17:27.912418 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:27.912393 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:17:31.776698 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:31.776665 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp" Oct 30 07:17:40.460596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:40.459933 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb"] Oct 30 07:17:40.483630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:40.483565 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:17:40.488892 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:40.488338 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb"] Oct 30 07:17:40.590978 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:40.590940 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxdcg\" (UniqueName: \"kubernetes.io/projected/745992af-9150-4336-bf13-2e0f087214a5-kube-api-access-xxdcg\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:17:40.591178 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:40.590992 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/745992af-9150-4336-bf13-2e0f087214a5-istiod-ca-cert\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:17:40.591178 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:40.591166 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-workload-certs\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:17:40.591291 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:40.591275 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-istio-envoy\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:17:40.591350 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:40.591338 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-workload-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:17:40.591392 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:40.591357 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-istio-data\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:17:40.591433 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:40.591402 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/745992af-9150-4336-bf13-2e0f087214a5-istio-podinfo\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:17:40.591504 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:40.591483 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/745992af-9150-4336-bf13-2e0f087214a5-istio-token\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:17:40.591694 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:40.591675 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-credential-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:17:40.693383 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:40.693350 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-credential-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:17:40.693535 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:40.693472 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxdcg\" (UniqueName: \"kubernetes.io/projected/745992af-9150-4336-bf13-2e0f087214a5-kube-api-access-xxdcg\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:17:40.693535 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:40.693491 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/745992af-9150-4336-bf13-2e0f087214a5-istiod-ca-cert\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:17:40.693696 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:40.693542 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-workload-certs\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:17:40.693696 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:40.693631 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-istio-envoy\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:17:40.693696 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:40.693690 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-workload-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:17:40.693858 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:40.693712 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-istio-data\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:17:40.693858 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:40.693752 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/745992af-9150-4336-bf13-2e0f087214a5-istio-podinfo\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:17:40.693858 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:40.693756 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-credential-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:17:40.693858 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:40.693825 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/745992af-9150-4336-bf13-2e0f087214a5-istio-token\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:17:40.694065 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:40.693887 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-workload-certs\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:17:40.694065 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:40.694048 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-istio-data\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:17:40.694318 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:40.694291 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/745992af-9150-4336-bf13-2e0f087214a5-istiod-ca-cert\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:17:40.694607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:40.694585 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-workload-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:17:40.695913 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:40.695886 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-istio-envoy\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:17:40.696235 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:40.696215 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/745992af-9150-4336-bf13-2e0f087214a5-istio-podinfo\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:17:40.743415 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:40.743301 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/745992af-9150-4336-bf13-2e0f087214a5-istio-token\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:17:40.753426 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:40.753335 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxdcg\" (UniqueName: \"kubernetes.io/projected/745992af-9150-4336-bf13-2e0f087214a5-kube-api-access-xxdcg\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:17:40.806056 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:40.806026 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:17:41.291594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:41.288325 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb"] Oct 30 07:17:41.458598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:41.454234 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:17:41.458598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:41.454360 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:17:41.458598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:41.454408 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:17:41.686183 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:41.686142 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" event={"ID":"745992af-9150-4336-bf13-2e0f087214a5","Type":"ContainerStarted","Data":"3343d0803013b4629c2128935e9174ec7feb1a7fefeede37759adb5a3284d980"} Oct 30 07:17:41.686729 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:41.686205 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" event={"ID":"745992af-9150-4336-bf13-2e0f087214a5","Type":"ContainerStarted","Data":"491b2835befd8efbb05f149b3907ec1e5e67ca046fb972bdc8e475513f1d798b"} Oct 30 07:17:41.749116 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:41.749041 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" podStartSLOduration=1.749021514 podStartE2EDuration="1.749021514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:17:41.748099371 +0000 UTC m=+1268.091147337" watchObservedRunningTime="2026-10-30 07:17:41.749021514 +0000 UTC m=+1268.092069481" Oct 30 07:17:41.899104 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:41.898998 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:17:42.814595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:42.813761 4951 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 07:17:42.814595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:42.813851 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" podUID="745992af-9150-4336-bf13-2e0f087214a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 07:17:42.915787 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:42.915752 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc"] Oct 30 07:17:42.916252 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:42.916223 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" podUID="2d36f333-0885-434c-aea5-0f9f1d44217c" containerName="istio-proxy" containerID="cri-o://2575f1f581e517ee9f67147ec3cdb8a10f15329e58bedea791bbc20cdcbe57ed" gracePeriod=30 Oct 30 07:17:43.894121 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:43.894080 4951 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 07:17:43.894606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:43.894147 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" podUID="745992af-9150-4336-bf13-2e0f087214a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 07:17:44.005589 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:44.005546 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f"] Oct 30 07:17:44.005976 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:44.005932 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" podUID="c5cb47c6-0296-4d55-8582-f7025fa4122d" containerName="istio-proxy" containerID="cri-o://efe7bf684de0aaea56de8f8d5aed614a62ff7de16ebaa88dbd4e9d8f83b7c425" gracePeriod=30 Oct 30 07:17:44.807114 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:44.807067 4951 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 07:17:44.807324 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:44.807136 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" podUID="745992af-9150-4336-bf13-2e0f087214a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 07:17:45.814534 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:45.807934 4951 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 07:17:45.814534 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:45.807995 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" podUID="745992af-9150-4336-bf13-2e0f087214a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 07:17:46.401786 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:46.401750 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2"] Oct 30 07:17:46.402372 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:46.402314 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" podUID="8a282e3a-aa71-4b0a-b9db-f475645f94e6" containerName="istio-proxy" containerID="cri-o://05a08253454e72f3d45a4bf4e2e2297f5bc0d311cccf2f1865b4106dcd38704a" gracePeriod=30 Oct 30 07:17:46.836944 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:46.832880 4951 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 07:17:46.837538 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:46.837492 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" podUID="745992af-9150-4336-bf13-2e0f087214a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 07:17:47.174315 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:47.174220 4951 patch_prober.go:29] interesting pod/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:17:47.174482 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:47.174306 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" podUID="2d36f333-0885-434c-aea5-0f9f1d44217c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:17:47.806772 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:47.806738 4951 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 07:17:47.807002 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:47.806979 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" podUID="745992af-9150-4336-bf13-2e0f087214a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 07:17:47.992093 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:47.992063 4951 generic.go:356] "Generic (PLEG): container finished" podID="2d36f333-0885-434c-aea5-0f9f1d44217c" containerID="2575f1f581e517ee9f67147ec3cdb8a10f15329e58bedea791bbc20cdcbe57ed" exitCode=0 Oct 30 07:17:47.992470 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:47.992132 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" event={"ID":"2d36f333-0885-434c-aea5-0f9f1d44217c","Type":"ContainerDied","Data":"2575f1f581e517ee9f67147ec3cdb8a10f15329e58bedea791bbc20cdcbe57ed"} Oct 30 07:17:47.992470 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:47.992218 4951 scope.go:122] "RemoveContainer" containerID="4946296448e1b6c3ff58d749bec9d611b7141e8fdc636827aeff859762826b71" Oct 30 07:17:48.371296 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:48.371219 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:17:48.402115 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:48.402077 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-workload-certs\") pod \"2d36f333-0885-434c-aea5-0f9f1d44217c\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " Oct 30 07:17:48.402303 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:48.402139 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-istio-data\") pod \"2d36f333-0885-434c-aea5-0f9f1d44217c\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " Oct 30 07:17:48.402303 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:48.402186 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d36f333-0885-434c-aea5-0f9f1d44217c-istio-token\" (UniqueName: \"kubernetes.io/projected/2d36f333-0885-434c-aea5-0f9f1d44217c-istio-token\") pod \"2d36f333-0885-434c-aea5-0f9f1d44217c\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " Oct 30 07:17:48.402303 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:48.402217 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d36f333-0885-434c-aea5-0f9f1d44217c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d36f333-0885-434c-aea5-0f9f1d44217c-istiod-ca-cert\") pod \"2d36f333-0885-434c-aea5-0f9f1d44217c\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " Oct 30 07:17:48.402469 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:48.402398 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-workload-certs" pod "2d36f333-0885-434c-aea5-0f9f1d44217c" (UID: "2d36f333-0885-434c-aea5-0f9f1d44217c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:17:48.402523 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:48.402482 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-credential-socket\") pod \"2d36f333-0885-434c-aea5-0f9f1d44217c\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " Oct 30 07:17:48.402601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:48.402520 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d36f333-0885-434c-aea5-0f9f1d44217c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d36f333-0885-434c-aea5-0f9f1d44217c-istio-podinfo\") pod \"2d36f333-0885-434c-aea5-0f9f1d44217c\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " Oct 30 07:17:48.402687 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:48.402664 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-istio-envoy\") pod \"2d36f333-0885-434c-aea5-0f9f1d44217c\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " Oct 30 07:17:48.402779 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:48.402762 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d36f333-0885-434c-aea5-0f9f1d44217c-istiod-ca-cert" pod "2d36f333-0885-434c-aea5-0f9f1d44217c" (UID: "2d36f333-0885-434c-aea5-0f9f1d44217c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:17:48.402841 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:48.402798 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d36f333-0885-434c-aea5-0f9f1d44217c-kube-api-access-zv26d\" (UniqueName: \"kubernetes.io/projected/2d36f333-0885-434c-aea5-0f9f1d44217c-kube-api-access-zv26d\") pod \"2d36f333-0885-434c-aea5-0f9f1d44217c\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " Oct 30 07:17:48.402899 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:48.402866 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-workload-socket\") pod \"2d36f333-0885-434c-aea5-0f9f1d44217c\" (UID: \"2d36f333-0885-434c-aea5-0f9f1d44217c\") " Oct 30 07:17:48.403595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:48.403288 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-istio-data" pod "2d36f333-0885-434c-aea5-0f9f1d44217c" (UID: "2d36f333-0885-434c-aea5-0f9f1d44217c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:17:48.403595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:48.403303 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-credential-socket" pod "2d36f333-0885-434c-aea5-0f9f1d44217c" (UID: "2d36f333-0885-434c-aea5-0f9f1d44217c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:17:48.404005 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:48.403905 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-workload-socket" pod "2d36f333-0885-434c-aea5-0f9f1d44217c" (UID: "2d36f333-0885-434c-aea5-0f9f1d44217c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:17:48.404193 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:48.404110 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:17:48.404193 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:48.404135 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:17:48.404193 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:48.404152 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:17:48.404193 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:48.404168 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:17:48.404193 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:48.404179 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d36f333-0885-434c-aea5-0f9f1d44217c-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:17:48.406127 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:48.405974 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d36f333-0885-434c-aea5-0f9f1d44217c-istio-podinfo" pod "2d36f333-0885-434c-aea5-0f9f1d44217c" (UID: "2d36f333-0885-434c-aea5-0f9f1d44217c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:17:48.406127 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:48.406102 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d36f333-0885-434c-aea5-0f9f1d44217c-istio-token" pod "2d36f333-0885-434c-aea5-0f9f1d44217c" (UID: "2d36f333-0885-434c-aea5-0f9f1d44217c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:17:48.406403 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:48.406375 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-istio-envoy" pod "2d36f333-0885-434c-aea5-0f9f1d44217c" (UID: "2d36f333-0885-434c-aea5-0f9f1d44217c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:17:48.407014 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:48.406883 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d36f333-0885-434c-aea5-0f9f1d44217c-kube-api-access-zv26d" pod "2d36f333-0885-434c-aea5-0f9f1d44217c" (UID: "2d36f333-0885-434c-aea5-0f9f1d44217c"). InnerVolumeSpecName "kube-api-access-zv26d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:17:48.516749 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:48.516700 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d36f333-0885-434c-aea5-0f9f1d44217c-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:17:48.517042 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:48.516980 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d36f333-0885-434c-aea5-0f9f1d44217c-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:17:48.517042 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:48.517011 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv26d\" (UniqueName: \"kubernetes.io/projected/2d36f333-0885-434c-aea5-0f9f1d44217c-kube-api-access-zv26d\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:17:48.517298 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:48.517224 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d36f333-0885-434c-aea5-0f9f1d44217c-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:17:48.807461 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:48.807426 4951 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 07:17:48.807727 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:48.807485 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" podUID="745992af-9150-4336-bf13-2e0f087214a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 07:17:48.957912 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:48.957839 4951 patch_prober.go:29] interesting pod/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:17:48.957912 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:48.957911 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" podUID="8a282e3a-aa71-4b0a-b9db-f475645f94e6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:17:49.004647 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.004614 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" event={"ID":"2d36f333-0885-434c-aea5-0f9f1d44217c","Type":"ContainerDied","Data":"e92f95ffa81c6d241d87307eaf062aa9ccf5b2f645ea441c56beb1e21d471600"} Oct 30 07:17:49.005006 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.004675 4951 scope.go:122] "RemoveContainer" containerID="2575f1f581e517ee9f67147ec3cdb8a10f15329e58bedea791bbc20cdcbe57ed" Oct 30 07:17:49.005006 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.004780 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc" Oct 30 07:17:49.099328 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.099266 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc"] Oct 30 07:17:49.103880 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.103853 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-7c51aa-openshift-default-56767768f7-r6wcc"] Oct 30 07:17:49.162909 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.162873 4951 patch_prober.go:29] interesting pod/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 07:17:49.163138 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.162940 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" podUID="c5cb47c6-0296-4d55-8582-f7025fa4122d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 07:17:49.520250 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.519886 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:17:49.581509 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.579465 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5cb47c6-0296-4d55-8582-f7025fa4122d-istio-token\" (UniqueName: \"kubernetes.io/projected/c5cb47c6-0296-4d55-8582-f7025fa4122d-istio-token\") pod \"c5cb47c6-0296-4d55-8582-f7025fa4122d\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " Oct 30 07:17:49.581509 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.579523 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-workload-socket\") pod \"c5cb47c6-0296-4d55-8582-f7025fa4122d\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " Oct 30 07:17:49.581509 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.579566 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-istio-data\") pod \"c5cb47c6-0296-4d55-8582-f7025fa4122d\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " Oct 30 07:17:49.581509 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.579714 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-workload-certs\") pod \"c5cb47c6-0296-4d55-8582-f7025fa4122d\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " Oct 30 07:17:49.581509 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.579842 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-credential-socket\") pod \"c5cb47c6-0296-4d55-8582-f7025fa4122d\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " Oct 30 07:17:49.581509 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.579925 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5cb47c6-0296-4d55-8582-f7025fa4122d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5cb47c6-0296-4d55-8582-f7025fa4122d-istio-podinfo\") pod \"c5cb47c6-0296-4d55-8582-f7025fa4122d\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " Oct 30 07:17:49.581509 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.579985 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5cb47c6-0296-4d55-8582-f7025fa4122d-kube-api-access-tv5wf\" (UniqueName: \"kubernetes.io/projected/c5cb47c6-0296-4d55-8582-f7025fa4122d-kube-api-access-tv5wf\") pod \"c5cb47c6-0296-4d55-8582-f7025fa4122d\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " Oct 30 07:17:49.581509 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.580086 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-istio-envoy\") pod \"c5cb47c6-0296-4d55-8582-f7025fa4122d\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " Oct 30 07:17:49.581509 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.580229 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5cb47c6-0296-4d55-8582-f7025fa4122d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5cb47c6-0296-4d55-8582-f7025fa4122d-istiod-ca-cert\") pod \"c5cb47c6-0296-4d55-8582-f7025fa4122d\" (UID: \"c5cb47c6-0296-4d55-8582-f7025fa4122d\") " Oct 30 07:17:49.581509 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.581431 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-workload-certs" pod "c5cb47c6-0296-4d55-8582-f7025fa4122d" (UID: "c5cb47c6-0296-4d55-8582-f7025fa4122d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:17:49.582665 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.582444 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-istio-data" pod "c5cb47c6-0296-4d55-8582-f7025fa4122d" (UID: "c5cb47c6-0296-4d55-8582-f7025fa4122d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:17:49.582871 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.582835 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-credential-socket" pod "c5cb47c6-0296-4d55-8582-f7025fa4122d" (UID: "c5cb47c6-0296-4d55-8582-f7025fa4122d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:17:49.583124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.583091 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5cb47c6-0296-4d55-8582-f7025fa4122d-istiod-ca-cert" pod "c5cb47c6-0296-4d55-8582-f7025fa4122d" (UID: "c5cb47c6-0296-4d55-8582-f7025fa4122d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:17:49.583399 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.583364 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-workload-socket" pod "c5cb47c6-0296-4d55-8582-f7025fa4122d" (UID: "c5cb47c6-0296-4d55-8582-f7025fa4122d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:17:49.586738 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.586696 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5cb47c6-0296-4d55-8582-f7025fa4122d-istio-podinfo" pod "c5cb47c6-0296-4d55-8582-f7025fa4122d" (UID: "c5cb47c6-0296-4d55-8582-f7025fa4122d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:17:49.587261 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.587216 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5cb47c6-0296-4d55-8582-f7025fa4122d-istio-token" pod "c5cb47c6-0296-4d55-8582-f7025fa4122d" (UID: "c5cb47c6-0296-4d55-8582-f7025fa4122d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:17:49.594173 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.594049 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5cb47c6-0296-4d55-8582-f7025fa4122d-kube-api-access-tv5wf" pod "c5cb47c6-0296-4d55-8582-f7025fa4122d" (UID: "c5cb47c6-0296-4d55-8582-f7025fa4122d"). InnerVolumeSpecName "kube-api-access-tv5wf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:17:49.594173 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.594146 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-istio-envoy" pod "c5cb47c6-0296-4d55-8582-f7025fa4122d" (UID: "c5cb47c6-0296-4d55-8582-f7025fa4122d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:17:49.682818 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.682767 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5cb47c6-0296-4d55-8582-f7025fa4122d-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:17:49.682818 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.682811 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:17:49.683078 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.682826 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:17:49.683078 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.682841 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:17:49.683078 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.682854 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:17:49.683078 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.682867 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5cb47c6-0296-4d55-8582-f7025fa4122d-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:17:49.683078 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.682882 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv5wf\" (UniqueName: \"kubernetes.io/projected/c5cb47c6-0296-4d55-8582-f7025fa4122d-kube-api-access-tv5wf\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:17:49.683078 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.682895 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5cb47c6-0296-4d55-8582-f7025fa4122d-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:17:49.683078 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.682908 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5cb47c6-0296-4d55-8582-f7025fa4122d-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:17:49.777734 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.777695 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d36f333-0885-434c-aea5-0f9f1d44217c" path="/var/lib/kubelet/pods/2d36f333-0885-434c-aea5-0f9f1d44217c/volumes" Oct 30 07:17:49.893942 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.893908 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:17:49.894853 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.894819 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:17:49.895698 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:49.895675 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:17:50.022758 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.022722 4951 generic.go:356] "Generic (PLEG): container finished" podID="c5cb47c6-0296-4d55-8582-f7025fa4122d" containerID="efe7bf684de0aaea56de8f8d5aed614a62ff7de16ebaa88dbd4e9d8f83b7c425" exitCode=0 Oct 30 07:17:50.023262 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.022829 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" event={"ID":"c5cb47c6-0296-4d55-8582-f7025fa4122d","Type":"ContainerDied","Data":"efe7bf684de0aaea56de8f8d5aed614a62ff7de16ebaa88dbd4e9d8f83b7c425"} Oct 30 07:17:50.023262 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.022859 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" Oct 30 07:17:50.023262 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.022885 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f" event={"ID":"c5cb47c6-0296-4d55-8582-f7025fa4122d","Type":"ContainerDied","Data":"d94db9c4296ce0ab9a61f4911bbd4d2e93ce8b3f061ee3c4b92dae487cf159f4"} Oct 30 07:17:50.023262 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.022918 4951 scope.go:122] "RemoveContainer" containerID="efe7bf684de0aaea56de8f8d5aed614a62ff7de16ebaa88dbd4e9d8f83b7c425" Oct 30 07:17:50.041872 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.041502 4951 scope.go:122] "RemoveContainer" containerID="d017f431005c40172c00b77017f518754017d3a71d0e3a548ef09c687749382b" Oct 30 07:17:50.059361 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.059339 4951 scope.go:122] "RemoveContainer" containerID="efe7bf684de0aaea56de8f8d5aed614a62ff7de16ebaa88dbd4e9d8f83b7c425" Oct 30 07:17:50.059856 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:17:50.059823 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efe7bf684de0aaea56de8f8d5aed614a62ff7de16ebaa88dbd4e9d8f83b7c425\": container with ID starting with efe7bf684de0aaea56de8f8d5aed614a62ff7de16ebaa88dbd4e9d8f83b7c425 not found: ID does not exist" containerID="efe7bf684de0aaea56de8f8d5aed614a62ff7de16ebaa88dbd4e9d8f83b7c425" Oct 30 07:17:50.059993 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.059869 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efe7bf684de0aaea56de8f8d5aed614a62ff7de16ebaa88dbd4e9d8f83b7c425"} err="failed to get container status \"efe7bf684de0aaea56de8f8d5aed614a62ff7de16ebaa88dbd4e9d8f83b7c425\": rpc error: code = NotFound desc = could not find container \"efe7bf684de0aaea56de8f8d5aed614a62ff7de16ebaa88dbd4e9d8f83b7c425\": container with ID starting with efe7bf684de0aaea56de8f8d5aed614a62ff7de16ebaa88dbd4e9d8f83b7c425 not found: ID does not exist" Oct 30 07:17:50.059993 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.059897 4951 scope.go:122] "RemoveContainer" containerID="d017f431005c40172c00b77017f518754017d3a71d0e3a548ef09c687749382b" Oct 30 07:17:50.060320 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:17:50.060288 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d017f431005c40172c00b77017f518754017d3a71d0e3a548ef09c687749382b\": container with ID starting with d017f431005c40172c00b77017f518754017d3a71d0e3a548ef09c687749382b not found: ID does not exist" containerID="d017f431005c40172c00b77017f518754017d3a71d0e3a548ef09c687749382b" Oct 30 07:17:50.060426 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.060341 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d017f431005c40172c00b77017f518754017d3a71d0e3a548ef09c687749382b"} err="failed to get container status \"d017f431005c40172c00b77017f518754017d3a71d0e3a548ef09c687749382b\": rpc error: code = NotFound desc = could not find container \"d017f431005c40172c00b77017f518754017d3a71d0e3a548ef09c687749382b\": container with ID starting with d017f431005c40172c00b77017f518754017d3a71d0e3a548ef09c687749382b not found: ID does not exist" Oct 30 07:17:50.079245 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.079210 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f"] Oct 30 07:17:50.084534 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.084502 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-jb28f"] Oct 30 07:17:50.375232 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.375146 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v"] Oct 30 07:17:50.376622 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.376591 4951 cpu_manager.go:414] "containerMap: removing container" podUID="2d36f333-0885-434c-aea5-0f9f1d44217c" containerName="istio-proxy" Oct 30 07:17:50.376795 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.376767 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d36f333-0885-434c-aea5-0f9f1d44217c" containerName="istio-proxy" Oct 30 07:17:50.376795 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.376798 4951 cpu_manager.go:414] "containerMap: removing container" podUID="2d36f333-0885-434c-aea5-0f9f1d44217c" containerName="istio-proxy" Oct 30 07:17:50.376976 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.376806 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d36f333-0885-434c-aea5-0f9f1d44217c" containerName="istio-proxy" Oct 30 07:17:50.376976 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.376877 4951 cpu_manager.go:414] "containerMap: removing container" podUID="c5cb47c6-0296-4d55-8582-f7025fa4122d" containerName="istio-proxy" Oct 30 07:17:50.376976 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.376884 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5cb47c6-0296-4d55-8582-f7025fa4122d" containerName="istio-proxy" Oct 30 07:17:50.376976 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.376922 4951 cpu_manager.go:414] "containerMap: removing container" podUID="c5cb47c6-0296-4d55-8582-f7025fa4122d" containerName="istio-proxy" Oct 30 07:17:50.376976 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.376929 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5cb47c6-0296-4d55-8582-f7025fa4122d" containerName="istio-proxy" Oct 30 07:17:50.377248 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.377193 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5cb47c6-0296-4d55-8582-f7025fa4122d" containerName="istio-proxy" Oct 30 07:17:50.377248 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.377214 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d36f333-0885-434c-aea5-0f9f1d44217c" containerName="istio-proxy" Oct 30 07:17:50.377248 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.377226 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d36f333-0885-434c-aea5-0f9f1d44217c" containerName="istio-proxy" Oct 30 07:17:50.383921 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.383893 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:17:50.406070 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.404527 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v"] Oct 30 07:17:50.409972 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.409904 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-486cd4-openshift-default-dockercfg-fgmb9\"" Oct 30 07:17:50.509739 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.509704 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istiod-ca-cert\") pod \"e2e-api-b-486cd4-openshift-default-7b44878777-zq44v\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:17:50.509936 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.509917 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istio-data\") pod \"e2e-api-b-486cd4-openshift-default-7b44878777-zq44v\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:17:50.509998 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.509958 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-workload-socket\") pod \"e2e-api-b-486cd4-openshift-default-7b44878777-zq44v\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:17:50.510113 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.510094 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqx8b\" (UniqueName: \"kubernetes.io/projected/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-kube-api-access-jqx8b\") pod \"e2e-api-b-486cd4-openshift-default-7b44878777-zq44v\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:17:50.510242 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.510223 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istio-token\") pod \"e2e-api-b-486cd4-openshift-default-7b44878777-zq44v\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:17:50.510298 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.510284 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-credential-socket\") pod \"e2e-api-b-486cd4-openshift-default-7b44878777-zq44v\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:17:50.510338 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.510311 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istio-envoy\") pod \"e2e-api-b-486cd4-openshift-default-7b44878777-zq44v\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:17:50.510432 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.510419 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-workload-certs\") pod \"e2e-api-b-486cd4-openshift-default-7b44878777-zq44v\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:17:50.510496 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.510454 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istio-podinfo\") pod \"e2e-api-b-486cd4-openshift-default-7b44878777-zq44v\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:17:50.638673 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.625957 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istio-podinfo\") pod \"e2e-api-b-486cd4-openshift-default-7b44878777-zq44v\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:17:50.638673 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.626042 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istiod-ca-cert\") pod \"e2e-api-b-486cd4-openshift-default-7b44878777-zq44v\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:17:50.638673 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.626210 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istio-data\") pod \"e2e-api-b-486cd4-openshift-default-7b44878777-zq44v\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:17:50.638673 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.626262 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-workload-socket\") pod \"e2e-api-b-486cd4-openshift-default-7b44878777-zq44v\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:17:50.638673 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.626402 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqx8b\" (UniqueName: \"kubernetes.io/projected/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-kube-api-access-jqx8b\") pod \"e2e-api-b-486cd4-openshift-default-7b44878777-zq44v\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:17:50.638673 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.626549 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istio-token\") pod \"e2e-api-b-486cd4-openshift-default-7b44878777-zq44v\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:17:50.638673 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.627215 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istio-data\") pod \"e2e-api-b-486cd4-openshift-default-7b44878777-zq44v\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:17:50.638673 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.627502 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-credential-socket\") pod \"e2e-api-b-486cd4-openshift-default-7b44878777-zq44v\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:17:50.638673 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.627538 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istio-envoy\") pod \"e2e-api-b-486cd4-openshift-default-7b44878777-zq44v\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:17:50.638673 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.627696 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-workload-certs\") pod \"e2e-api-b-486cd4-openshift-default-7b44878777-zq44v\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:17:50.638673 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.628073 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-workload-certs\") pod \"e2e-api-b-486cd4-openshift-default-7b44878777-zq44v\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:17:50.638673 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.629452 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istiod-ca-cert\") pod \"e2e-api-b-486cd4-openshift-default-7b44878777-zq44v\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:17:50.638673 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.629701 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-workload-socket\") pod \"e2e-api-b-486cd4-openshift-default-7b44878777-zq44v\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:17:50.638673 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.629923 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-credential-socket\") pod \"e2e-api-b-486cd4-openshift-default-7b44878777-zq44v\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:17:50.663596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.662534 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istio-envoy\") pod \"e2e-api-b-486cd4-openshift-default-7b44878777-zq44v\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:17:50.711322 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.689335 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqx8b\" (UniqueName: \"kubernetes.io/projected/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-kube-api-access-jqx8b\") pod \"e2e-api-b-486cd4-openshift-default-7b44878777-zq44v\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:17:50.711322 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.699441 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istio-podinfo\") pod \"e2e-api-b-486cd4-openshift-default-7b44878777-zq44v\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:17:50.711322 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.711227 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istio-token\") pod \"e2e-api-b-486cd4-openshift-default-7b44878777-zq44v\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:17:50.727357 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:50.727040 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:17:51.288315 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:51.288284 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v"] Oct 30 07:17:51.289074 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:17:51.289042 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3d3f735_6cd0_4153_a6ff_88f77f4941ca.slice/crio-8e70637230d66cb975a6362fab7643d6529641d214ef248fdd1490ffc1376224 WatchSource:0}: Error finding container 8e70637230d66cb975a6362fab7643d6529641d214ef248fdd1490ffc1376224: Status 404 returned error can't find the container with id 8e70637230d66cb975a6362fab7643d6529641d214ef248fdd1490ffc1376224 Oct 30 07:17:51.293415 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:51.293380 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:17:51.293500 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:51.293449 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:17:51.293500 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:51.293479 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:17:52.149061 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.149007 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5cb47c6-0296-4d55-8582-f7025fa4122d" path="/var/lib/kubelet/pods/c5cb47c6-0296-4d55-8582-f7025fa4122d/volumes" Oct 30 07:17:52.168222 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.168113 4951 generic.go:356] "Generic (PLEG): container finished" podID="8a282e3a-aa71-4b0a-b9db-f475645f94e6" containerID="05a08253454e72f3d45a4bf4e2e2297f5bc0d311cccf2f1865b4106dcd38704a" exitCode=0 Oct 30 07:17:52.168397 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.168217 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" event={"ID":"8a282e3a-aa71-4b0a-b9db-f475645f94e6","Type":"ContainerDied","Data":"05a08253454e72f3d45a4bf4e2e2297f5bc0d311cccf2f1865b4106dcd38704a"} Oct 30 07:17:52.168397 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.168299 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" event={"ID":"8a282e3a-aa71-4b0a-b9db-f475645f94e6","Type":"ContainerDied","Data":"e19be68153f4ae36d3d33f98d0478eabdf97c72963da40d162b1f3a24b8048b8"} Oct 30 07:17:52.168397 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.168333 4951 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e19be68153f4ae36d3d33f98d0478eabdf97c72963da40d162b1f3a24b8048b8" Oct 30 07:17:52.168397 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.168351 4951 scope.go:122] "RemoveContainer" containerID="993ee6f9224afbd09b035fb21c698416775eddccf8f94a896ef1e155d191c172" Oct 30 07:17:52.170095 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.170073 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:17:52.172122 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.172088 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" event={"ID":"f3d3f735-6cd0-4153-a6ff-88f77f4941ca","Type":"ContainerStarted","Data":"27a16ce17a1d15ea1f14c838de0bb4c00094047037482fb1b05dab8f0bbebc87"} Oct 30 07:17:52.172247 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.172155 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" event={"ID":"f3d3f735-6cd0-4153-a6ff-88f77f4941ca","Type":"ContainerStarted","Data":"8e70637230d66cb975a6362fab7643d6529641d214ef248fdd1490ffc1376224"} Oct 30 07:17:52.236770 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.236279 4951 patch_prober.go:29] interesting pod/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": read tcp 10.132.0.2:46012->10.132.0.103:15021: read: connection reset by peer" start-of-body= Oct 30 07:17:52.236770 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.236352 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" podUID="6cba7319-64a7-453f-95c7-ccf0c18033a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": read tcp 10.132.0.2:46012->10.132.0.103:15021: read: connection reset by peer" Oct 30 07:17:52.251356 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.251320 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istio-podinfo\") pod \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " Oct 30 07:17:52.251552 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.251431 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-workload-socket\") pod \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " Oct 30 07:17:52.251844 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.251814 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-workload-socket" pod "8a282e3a-aa71-4b0a-b9db-f475645f94e6" (UID: "8a282e3a-aa71-4b0a-b9db-f475645f94e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:17:52.251970 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.251638 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a282e3a-aa71-4b0a-b9db-f475645f94e6-kube-api-access-gnsxv\" (UniqueName: \"kubernetes.io/projected/8a282e3a-aa71-4b0a-b9db-f475645f94e6-kube-api-access-gnsxv\") pod \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " Oct 30 07:17:52.252034 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.251992 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istiod-ca-cert\") pod \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " Oct 30 07:17:52.252034 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.252022 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istio-envoy\") pod \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " Oct 30 07:17:52.252334 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.252278 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istio-data\") pod \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " Oct 30 07:17:52.252824 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.252801 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-workload-certs\") pod \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " Oct 30 07:17:52.252972 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.252952 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istio-token\" (UniqueName: \"kubernetes.io/projected/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istio-token\") pod \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " Oct 30 07:17:52.253067 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.253041 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-credential-socket\") pod \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\" (UID: \"8a282e3a-aa71-4b0a-b9db-f475645f94e6\") " Oct 30 07:17:52.253310 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.253261 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istio-data" pod "8a282e3a-aa71-4b0a-b9db-f475645f94e6" (UID: "8a282e3a-aa71-4b0a-b9db-f475645f94e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:17:52.253505 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.253485 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-workload-certs" pod "8a282e3a-aa71-4b0a-b9db-f475645f94e6" (UID: "8a282e3a-aa71-4b0a-b9db-f475645f94e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:17:52.254031 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.254007 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:17:52.254200 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.254185 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:17:52.254324 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.254311 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:17:52.254409 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.254024 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istiod-ca-cert" pod "8a282e3a-aa71-4b0a-b9db-f475645f94e6" (UID: "8a282e3a-aa71-4b0a-b9db-f475645f94e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:17:52.256126 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.256101 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-credential-socket" pod "8a282e3a-aa71-4b0a-b9db-f475645f94e6" (UID: "8a282e3a-aa71-4b0a-b9db-f475645f94e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:17:52.260052 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.260025 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istio-token" pod "8a282e3a-aa71-4b0a-b9db-f475645f94e6" (UID: "8a282e3a-aa71-4b0a-b9db-f475645f94e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:17:52.269653 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.269553 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" podStartSLOduration=2.269535925 podStartE2EDuration="2.269535925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:17:52.26827142 +0000 UTC m=+1278.611319422" watchObservedRunningTime="2026-10-30 07:17:52.269535925 +0000 UTC m=+1278.612583892" Oct 30 07:17:52.269903 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.269877 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istio-envoy" pod "8a282e3a-aa71-4b0a-b9db-f475645f94e6" (UID: "8a282e3a-aa71-4b0a-b9db-f475645f94e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:17:52.276888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.276863 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a282e3a-aa71-4b0a-b9db-f475645f94e6-kube-api-access-gnsxv" pod "8a282e3a-aa71-4b0a-b9db-f475645f94e6" (UID: "8a282e3a-aa71-4b0a-b9db-f475645f94e6"). InnerVolumeSpecName "kube-api-access-gnsxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:17:52.278648 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.278561 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istio-podinfo" pod "8a282e3a-aa71-4b0a-b9db-f475645f94e6" (UID: "8a282e3a-aa71-4b0a-b9db-f475645f94e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:17:52.356623 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.356471 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:17:52.356623 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.356508 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:17:52.356623 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.356522 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:17:52.356623 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.356536 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnsxv\" (UniqueName: \"kubernetes.io/projected/8a282e3a-aa71-4b0a-b9db-f475645f94e6-kube-api-access-gnsxv\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:17:52.356623 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.356548 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:17:52.356623 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.356560 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a282e3a-aa71-4b0a-b9db-f475645f94e6-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:17:52.738644 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.728032 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:17:52.757598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:52.753898 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:17:53.024771 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:17:53.024730 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:17:53.181365 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:53.181331 4951 generic.go:356] "Generic (PLEG): container finished" podID="6cba7319-64a7-453f-95c7-ccf0c18033a0" containerID="15a6be79554d14c24357a6dc1e551cb55affdfada740b82fee4d6b1982bdb138" exitCode=0 Oct 30 07:17:53.181547 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:53.181407 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" event={"ID":"6cba7319-64a7-453f-95c7-ccf0c18033a0","Type":"ContainerDied","Data":"15a6be79554d14c24357a6dc1e551cb55affdfada740b82fee4d6b1982bdb138"} Oct 30 07:17:53.182005 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:53.181985 4951 scope.go:122] "RemoveContainer" containerID="15a6be79554d14c24357a6dc1e551cb55affdfada740b82fee4d6b1982bdb138" Oct 30 07:17:53.183275 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:53.183259 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2" Oct 30 07:17:53.183652 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:53.183604 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:17:53.184023 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:53.183987 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:17:53.184104 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:53.184078 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:17:53.184144 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:53.184121 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:17:53.184937 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:53.184859 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:17:53.295253 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:53.295226 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2"] Oct 30 07:17:53.301481 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:53.301447 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-77ef52-openshift-default-589d56f4cc-lw5s2"] Oct 30 07:17:53.781414 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:53.781380 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a282e3a-aa71-4b0a-b9db-f475645f94e6" path="/var/lib/kubelet/pods/8a282e3a-aa71-4b0a-b9db-f475645f94e6/volumes" Oct 30 07:17:54.185738 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:54.185633 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-56eb7f-5fbf45d446-d6gzv"] Oct 30 07:17:54.193371 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:54.193339 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" event={"ID":"6cba7319-64a7-453f-95c7-ccf0c18033a0","Type":"ContainerStarted","Data":"44d32d3e5b94f77719d68955c14e42cef14f989a5db5bd0b064e1f33b23d0846"} Oct 30 07:17:55.099490 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:55.097085 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:17:55.099490 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:55.097237 4951 patch_prober.go:29] interesting pod/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 07:17:55.099490 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:55.097283 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" podUID="6cba7319-64a7-453f-95c7-ccf0c18033a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 07:17:56.097751 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:56.097711 4951 patch_prober.go:29] interesting pod/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 07:17:56.097969 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:56.097789 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" podUID="6cba7319-64a7-453f-95c7-ccf0c18033a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 07:17:56.722622 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:56.722075 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-486cd4-5758d6dfc9-v5nfj"] Oct 30 07:17:56.736596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:56.733263 4951 cpu_manager.go:414] "containerMap: removing container" podUID="8a282e3a-aa71-4b0a-b9db-f475645f94e6" containerName="istio-proxy" Oct 30 07:17:56.736596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:56.733292 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a282e3a-aa71-4b0a-b9db-f475645f94e6" containerName="istio-proxy" Oct 30 07:17:56.736596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:56.733539 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a282e3a-aa71-4b0a-b9db-f475645f94e6" containerName="istio-proxy" Oct 30 07:17:56.736596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:56.733559 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a282e3a-aa71-4b0a-b9db-f475645f94e6" containerName="istio-proxy" Oct 30 07:17:56.736596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:56.733593 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5cb47c6-0296-4d55-8582-f7025fa4122d" containerName="istio-proxy" Oct 30 07:17:56.751604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:56.750880 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-486cd4-5758d6dfc9-v5nfj" Oct 30 07:17:56.752672 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:56.752646 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-486cd4-5758d6dfc9-v5nfj"] Oct 30 07:17:56.809016 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:56.808977 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-486cd4-5758d6dfc9-v5nfj\" (UID: \"8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-486cd4-5758d6dfc9-v5nfj" Oct 30 07:17:56.809445 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:56.809413 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwhkg\" (UniqueName: \"kubernetes.io/projected/8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1-kube-api-access-gwhkg\") pod \"payload-pre-processing-e2e-api-b-486cd4-5758d6dfc9-v5nfj\" (UID: \"8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-486cd4-5758d6dfc9-v5nfj" Oct 30 07:17:56.832227 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:56.832194 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-486cd4-dockercfg-m6j9m\"" Oct 30 07:17:56.861676 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:56.832532 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-486cd4\"" Oct 30 07:17:56.974711 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:56.931858 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-486cd4-5758d6dfc9-v5nfj\" (UID: \"8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-486cd4-5758d6dfc9-v5nfj" Oct 30 07:17:56.974711 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:56.932062 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwhkg\" (UniqueName: \"kubernetes.io/projected/8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1-kube-api-access-gwhkg\") pod \"payload-pre-processing-e2e-api-b-486cd4-5758d6dfc9-v5nfj\" (UID: \"8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-486cd4-5758d6dfc9-v5nfj" Oct 30 07:17:57.075854 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:56.975263 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-486cd4-5758d6dfc9-v5nfj\" (UID: \"8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-486cd4-5758d6dfc9-v5nfj" Oct 30 07:17:57.075854 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:57.032044 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwhkg\" (UniqueName: \"kubernetes.io/projected/8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1-kube-api-access-gwhkg\") pod \"payload-pre-processing-e2e-api-b-486cd4-5758d6dfc9-v5nfj\" (UID: \"8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-486cd4-5758d6dfc9-v5nfj" Oct 30 07:17:57.105045 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:57.105008 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-486cd4-5db6fd58-tg4wr"] Oct 30 07:17:57.106506 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:57.106474 4951 cpu_manager.go:414] "containerMap: removing container" podUID="8a282e3a-aa71-4b0a-b9db-f475645f94e6" containerName="istio-proxy" Oct 30 07:17:57.112367 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:57.112335 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a282e3a-aa71-4b0a-b9db-f475645f94e6" containerName="istio-proxy" Oct 30 07:17:57.114878 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:57.114848 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-486cd4-5758d6dfc9-v5nfj" Oct 30 07:17:57.257095 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:57.257064 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-486cd4-5db6fd58-tg4wr" Oct 30 07:17:57.275094 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:57.275061 4951 patch_prober.go:29] interesting pod/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 07:17:57.275425 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:57.275387 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" podUID="6cba7319-64a7-453f-95c7-ccf0c18033a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 07:17:57.325627 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:57.312275 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-486cd4-5db6fd58-tg4wr"] Oct 30 07:17:57.364715 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:57.364677 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10fee9bf-b760-42ee-bd0c-93a44cee033e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-486cd4-5db6fd58-tg4wr\" (UID: \"10fee9bf-b760-42ee-bd0c-93a44cee033e\") " pod="openshift-ingress/payload-processing-e2e-api-b-486cd4-5db6fd58-tg4wr" Oct 30 07:17:57.394611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:57.391026 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n9mr\" (UniqueName: \"kubernetes.io/projected/10fee9bf-b760-42ee-bd0c-93a44cee033e-kube-api-access-8n9mr\") pod \"payload-processing-e2e-api-b-486cd4-5db6fd58-tg4wr\" (UID: \"10fee9bf-b760-42ee-bd0c-93a44cee033e\") " pod="openshift-ingress/payload-processing-e2e-api-b-486cd4-5db6fd58-tg4wr" Oct 30 07:17:57.420598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:57.390892 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1e3dd1-789c944fd4-m74hx"] Oct 30 07:17:57.470643 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:57.435731 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1e3dd1-695f9f69f9-w54sq"] Oct 30 07:17:57.528643 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:57.503046 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10fee9bf-b760-42ee-bd0c-93a44cee033e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-486cd4-5db6fd58-tg4wr\" (UID: \"10fee9bf-b760-42ee-bd0c-93a44cee033e\") " pod="openshift-ingress/payload-processing-e2e-api-b-486cd4-5db6fd58-tg4wr" Oct 30 07:17:57.528643 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:57.503126 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n9mr\" (UniqueName: \"kubernetes.io/projected/10fee9bf-b760-42ee-bd0c-93a44cee033e-kube-api-access-8n9mr\") pod \"payload-processing-e2e-api-b-486cd4-5db6fd58-tg4wr\" (UID: \"10fee9bf-b760-42ee-bd0c-93a44cee033e\") " pod="openshift-ingress/payload-processing-e2e-api-b-486cd4-5db6fd58-tg4wr" Oct 30 07:17:57.528643 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:57.504703 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10fee9bf-b760-42ee-bd0c-93a44cee033e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-486cd4-5db6fd58-tg4wr\" (UID: \"10fee9bf-b760-42ee-bd0c-93a44cee033e\") " pod="openshift-ingress/payload-processing-e2e-api-b-486cd4-5db6fd58-tg4wr" Oct 30 07:17:57.603874 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:57.603840 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n9mr\" (UniqueName: \"kubernetes.io/projected/10fee9bf-b760-42ee-bd0c-93a44cee033e-kube-api-access-8n9mr\") pod \"payload-processing-e2e-api-b-486cd4-5db6fd58-tg4wr\" (UID: \"10fee9bf-b760-42ee-bd0c-93a44cee033e\") " pod="openshift-ingress/payload-processing-e2e-api-b-486cd4-5db6fd58-tg4wr" Oct 30 07:17:57.688710 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:57.680807 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-486cd4-5db6fd58-tg4wr" Oct 30 07:17:58.107807 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:58.093750 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-486cd4-5758d6dfc9-v5nfj"] Oct 30 07:17:58.161597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:58.131012 4951 patch_prober.go:29] interesting pod/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 07:17:58.161597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:58.131077 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" podUID="6cba7319-64a7-453f-95c7-ccf0c18033a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 07:17:58.313665 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:58.313632 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-486cd4-5758d6dfc9-v5nfj" event={"ID":"8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1","Type":"ContainerStarted","Data":"157ad8f9be2f8b7ddb1b98c068fe70ac6e094661a9304c6e782528f4848f48e0"} Oct 30 07:17:58.499454 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:58.499429 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-486cd4-5db6fd58-tg4wr"] Oct 30 07:17:59.097823 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:59.097775 4951 patch_prober.go:29] interesting pod/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 07:17:59.098007 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:59.097849 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" podUID="6cba7319-64a7-453f-95c7-ccf0c18033a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 07:17:59.186185 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:59.186131 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-56eb7f-5fbf45d446-d6gzv" podUID="1e764719-1b26-4ba1-87ba-fe5499e58071" containerName="payload-processing" containerID="cri-o://3cad1a0b4b107577b15a9aca2b2893acbf6f65cd9b8ef562a973f70bc8d713fd" gracePeriod=25 Oct 30 07:17:59.353651 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:59.352781 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-486cd4-5758d6dfc9-v5nfj" event={"ID":"8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1","Type":"ContainerStarted","Data":"c0541eafaad5dae72820871097d23c1509ce657ef20fc4ee509f6dc2910abd58"} Oct 30 07:17:59.353651 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:59.352875 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-486cd4-5758d6dfc9-v5nfj" Oct 30 07:17:59.356190 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:59.356157 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-486cd4-5db6fd58-tg4wr" event={"ID":"10fee9bf-b760-42ee-bd0c-93a44cee033e","Type":"ContainerStarted","Data":"1714b0aebfb2a59dbead75147b9edf8e81362625c97b77313291e98c1a6089a2"} Oct 30 07:17:59.356331 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:59.356228 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-486cd4-5db6fd58-tg4wr" Oct 30 07:17:59.356331 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:59.356253 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-486cd4-5db6fd58-tg4wr" event={"ID":"10fee9bf-b760-42ee-bd0c-93a44cee033e","Type":"ContainerStarted","Data":"0bfb00d72f1efb00d7846ebf2dc6b11d26f347c91e421765a82f02ed4691349f"} Oct 30 07:17:59.358555 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:59.358527 4951 generic.go:356] "Generic (PLEG): container finished" podID="1e764719-1b26-4ba1-87ba-fe5499e58071" containerID="3cad1a0b4b107577b15a9aca2b2893acbf6f65cd9b8ef562a973f70bc8d713fd" exitCode=0 Oct 30 07:17:59.358780 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:59.358763 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-56eb7f-5fbf45d446-d6gzv" event={"ID":"1e764719-1b26-4ba1-87ba-fe5499e58071","Type":"ContainerDied","Data":"3cad1a0b4b107577b15a9aca2b2893acbf6f65cd9b8ef562a973f70bc8d713fd"} Oct 30 07:17:59.393472 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:59.393418 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-486cd4-5758d6dfc9-v5nfj" podStartSLOduration=3.3934031669999998 podStartE2EDuration="3.393403167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:17:59.389884127 +0000 UTC m=+1285.732932095" watchObservedRunningTime="2026-10-30 07:17:59.393403167 +0000 UTC m=+1285.736451171" Oct 30 07:17:59.867609 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:59.867183 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-56eb7f-5fbf45d446-d6gzv" Oct 30 07:17:59.907936 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:59.907883 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-486cd4-5db6fd58-tg4wr" podStartSLOduration=3.907865122 podStartE2EDuration="3.907865122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:17:59.487937054 +0000 UTC m=+1285.830985033" watchObservedRunningTime="2026-10-30 07:17:59.907865122 +0000 UTC m=+1286.250913086" Oct 30 07:17:59.926608 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:59.923753 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e764719-1b26-4ba1-87ba-fe5499e58071-kube-api-access-glpmk\" (UniqueName: \"kubernetes.io/projected/1e764719-1b26-4ba1-87ba-fe5499e58071-kube-api-access-glpmk\") pod \"1e764719-1b26-4ba1-87ba-fe5499e58071\" (UID: \"1e764719-1b26-4ba1-87ba-fe5499e58071\") " Oct 30 07:17:59.926608 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:59.924031 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e764719-1b26-4ba1-87ba-fe5499e58071-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e764719-1b26-4ba1-87ba-fe5499e58071-plugins-config-volume\") pod \"1e764719-1b26-4ba1-87ba-fe5499e58071\" (UID: \"1e764719-1b26-4ba1-87ba-fe5499e58071\") " Oct 30 07:17:59.945285 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:59.945243 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e764719-1b26-4ba1-87ba-fe5499e58071-plugins-config-volume" pod "1e764719-1b26-4ba1-87ba-fe5499e58071" (UID: "1e764719-1b26-4ba1-87ba-fe5499e58071"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:17:59.974296 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:17:59.974260 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e764719-1b26-4ba1-87ba-fe5499e58071-kube-api-access-glpmk" pod "1e764719-1b26-4ba1-87ba-fe5499e58071" (UID: "1e764719-1b26-4ba1-87ba-fe5499e58071"). InnerVolumeSpecName "kube-api-access-glpmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:00.046499 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:00.046457 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e764719-1b26-4ba1-87ba-fe5499e58071-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:00.046499 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:00.046491 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glpmk\" (UniqueName: \"kubernetes.io/projected/1e764719-1b26-4ba1-87ba-fe5499e58071-kube-api-access-glpmk\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:00.097217 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:00.097129 4951 patch_prober.go:29] interesting pod/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 07:18:00.097217 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:00.097187 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" podUID="6cba7319-64a7-453f-95c7-ccf0c18033a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 07:18:00.368921 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:00.368823 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-56eb7f-5fbf45d446-d6gzv" event={"ID":"1e764719-1b26-4ba1-87ba-fe5499e58071","Type":"ContainerDied","Data":"74f5828bdc4632591d3637acbe1e0a4f21029d67ff2df873af46915eefba8ba7"} Oct 30 07:18:00.368921 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:00.368843 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-56eb7f-5fbf45d446-d6gzv" Oct 30 07:18:00.368921 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:00.368903 4951 scope.go:122] "RemoveContainer" containerID="3cad1a0b4b107577b15a9aca2b2893acbf6f65cd9b8ef562a973f70bc8d713fd" Oct 30 07:18:00.422244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:00.422191 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-56eb7f-5fbf45d446-d6gzv"] Oct 30 07:18:00.426159 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:00.426126 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-56eb7f-5fbf45d446-d6gzv"] Oct 30 07:18:01.101841 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:01.101812 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:18:01.102109 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:01.102088 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:18:01.102892 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:01.102862 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:18:01.776983 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:01.776951 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e764719-1b26-4ba1-87ba-fe5499e58071" path="/var/lib/kubelet/pods/1e764719-1b26-4ba1-87ba-fe5499e58071/volumes" Oct 30 07:18:02.436566 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:02.436514 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-1e3dd1-789c944fd4-m74hx" podUID="fc1e4717-80e7-47d7-b7f7-75a9a2477d50" containerName="payload-processing" containerID="cri-o://10e1c51dda675c0a786c24efd613f4540a95953278a436468ff426c7ae51d938" gracePeriod=25 Oct 30 07:18:02.436774 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:02.436527 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e3dd1-695f9f69f9-w54sq" podUID="d2d13e7a-5d8a-417f-aff7-746e41ba4510" containerName="payload-pre-processing" containerID="cri-o://2055a75c086474ec00a0145cbe9d3c1470101879757142ced5357933e15f7d81" gracePeriod=25 Oct 30 07:18:02.759613 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:02.756107 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e3dd1-695f9f69f9-w54sq" Oct 30 07:18:02.814554 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:02.814520 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1e3dd1-789c944fd4-m74hx" Oct 30 07:18:02.839761 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:02.839712 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2d13e7a-5d8a-417f-aff7-746e41ba4510-kube-api-access-jwj7c\" (UniqueName: \"kubernetes.io/projected/d2d13e7a-5d8a-417f-aff7-746e41ba4510-kube-api-access-jwj7c\") pod \"d2d13e7a-5d8a-417f-aff7-746e41ba4510\" (UID: \"d2d13e7a-5d8a-417f-aff7-746e41ba4510\") " Oct 30 07:18:02.839943 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:02.839766 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2d13e7a-5d8a-417f-aff7-746e41ba4510-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d13e7a-5d8a-417f-aff7-746e41ba4510-plugins-config-volume\") pod \"d2d13e7a-5d8a-417f-aff7-746e41ba4510\" (UID: \"d2d13e7a-5d8a-417f-aff7-746e41ba4510\") " Oct 30 07:18:02.840457 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:02.840429 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2d13e7a-5d8a-417f-aff7-746e41ba4510-plugins-config-volume" pod "d2d13e7a-5d8a-417f-aff7-746e41ba4510" (UID: "d2d13e7a-5d8a-417f-aff7-746e41ba4510"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:18:02.841158 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:02.841135 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d13e7a-5d8a-417f-aff7-746e41ba4510-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:02.846101 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:02.846073 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2d13e7a-5d8a-417f-aff7-746e41ba4510-kube-api-access-jwj7c" pod "d2d13e7a-5d8a-417f-aff7-746e41ba4510" (UID: "d2d13e7a-5d8a-417f-aff7-746e41ba4510"). InnerVolumeSpecName "kube-api-access-jwj7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:02.942005 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:02.941974 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc1e4717-80e7-47d7-b7f7-75a9a2477d50-kube-api-access-7v98b\" (UniqueName: \"kubernetes.io/projected/fc1e4717-80e7-47d7-b7f7-75a9a2477d50-kube-api-access-7v98b\") pod \"fc1e4717-80e7-47d7-b7f7-75a9a2477d50\" (UID: \"fc1e4717-80e7-47d7-b7f7-75a9a2477d50\") " Oct 30 07:18:02.942179 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:02.942146 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc1e4717-80e7-47d7-b7f7-75a9a2477d50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1e4717-80e7-47d7-b7f7-75a9a2477d50-plugins-config-volume\") pod \"fc1e4717-80e7-47d7-b7f7-75a9a2477d50\" (UID: \"fc1e4717-80e7-47d7-b7f7-75a9a2477d50\") " Oct 30 07:18:02.942585 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:02.942530 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc1e4717-80e7-47d7-b7f7-75a9a2477d50-plugins-config-volume" pod "fc1e4717-80e7-47d7-b7f7-75a9a2477d50" (UID: "fc1e4717-80e7-47d7-b7f7-75a9a2477d50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:18:02.942974 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:02.942935 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwj7c\" (UniqueName: \"kubernetes.io/projected/d2d13e7a-5d8a-417f-aff7-746e41ba4510-kube-api-access-jwj7c\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:02.942974 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:02.942974 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1e4717-80e7-47d7-b7f7-75a9a2477d50-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:02.944379 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:02.944353 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc1e4717-80e7-47d7-b7f7-75a9a2477d50-kube-api-access-7v98b" pod "fc1e4717-80e7-47d7-b7f7-75a9a2477d50" (UID: "fc1e4717-80e7-47d7-b7f7-75a9a2477d50"). InnerVolumeSpecName "kube-api-access-7v98b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:03.044660 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:03.044621 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v98b\" (UniqueName: \"kubernetes.io/projected/fc1e4717-80e7-47d7-b7f7-75a9a2477d50-kube-api-access-7v98b\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:03.412139 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:03.412054 4951 generic.go:356] "Generic (PLEG): container finished" podID="d2d13e7a-5d8a-417f-aff7-746e41ba4510" containerID="2055a75c086474ec00a0145cbe9d3c1470101879757142ced5357933e15f7d81" exitCode=0 Oct 30 07:18:03.412139 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:03.412107 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e3dd1-695f9f69f9-w54sq" Oct 30 07:18:03.412384 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:03.412141 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e3dd1-695f9f69f9-w54sq" event={"ID":"d2d13e7a-5d8a-417f-aff7-746e41ba4510","Type":"ContainerDied","Data":"2055a75c086474ec00a0145cbe9d3c1470101879757142ced5357933e15f7d81"} Oct 30 07:18:03.412384 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:03.412202 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e3dd1-695f9f69f9-w54sq" event={"ID":"d2d13e7a-5d8a-417f-aff7-746e41ba4510","Type":"ContainerDied","Data":"044434dfc32467f2d139cf6a807ccc40c76f3f27f59347512f12f439e80354ed"} Oct 30 07:18:03.412384 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:03.412234 4951 scope.go:122] "RemoveContainer" containerID="2055a75c086474ec00a0145cbe9d3c1470101879757142ced5357933e15f7d81" Oct 30 07:18:03.413690 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:03.413666 4951 generic.go:356] "Generic (PLEG): container finished" podID="fc1e4717-80e7-47d7-b7f7-75a9a2477d50" containerID="10e1c51dda675c0a786c24efd613f4540a95953278a436468ff426c7ae51d938" exitCode=0 Oct 30 07:18:03.413815 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:03.413712 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1e3dd1-789c944fd4-m74hx" Oct 30 07:18:03.413815 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:03.413736 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1e3dd1-789c944fd4-m74hx" event={"ID":"fc1e4717-80e7-47d7-b7f7-75a9a2477d50","Type":"ContainerDied","Data":"10e1c51dda675c0a786c24efd613f4540a95953278a436468ff426c7ae51d938"} Oct 30 07:18:03.413815 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:03.413781 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1e3dd1-789c944fd4-m74hx" event={"ID":"fc1e4717-80e7-47d7-b7f7-75a9a2477d50","Type":"ContainerDied","Data":"393d1623862f095403bb623de5c93890362643c853c1a8c518b7a94cd163f391"} Oct 30 07:18:03.424790 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:03.424771 4951 scope.go:122] "RemoveContainer" containerID="2055a75c086474ec00a0145cbe9d3c1470101879757142ced5357933e15f7d81" Oct 30 07:18:03.425139 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:18:03.425115 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2055a75c086474ec00a0145cbe9d3c1470101879757142ced5357933e15f7d81\": container with ID starting with 2055a75c086474ec00a0145cbe9d3c1470101879757142ced5357933e15f7d81 not found: ID does not exist" containerID="2055a75c086474ec00a0145cbe9d3c1470101879757142ced5357933e15f7d81" Oct 30 07:18:03.425241 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:03.425146 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2055a75c086474ec00a0145cbe9d3c1470101879757142ced5357933e15f7d81"} err="failed to get container status \"2055a75c086474ec00a0145cbe9d3c1470101879757142ced5357933e15f7d81\": rpc error: code = NotFound desc = could not find container \"2055a75c086474ec00a0145cbe9d3c1470101879757142ced5357933e15f7d81\": container with ID starting with 2055a75c086474ec00a0145cbe9d3c1470101879757142ced5357933e15f7d81 not found: ID does not exist" Oct 30 07:18:03.425241 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:03.425165 4951 scope.go:122] "RemoveContainer" containerID="10e1c51dda675c0a786c24efd613f4540a95953278a436468ff426c7ae51d938" Oct 30 07:18:03.436872 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:03.436846 4951 scope.go:122] "RemoveContainer" containerID="10e1c51dda675c0a786c24efd613f4540a95953278a436468ff426c7ae51d938" Oct 30 07:18:03.437193 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:18:03.437173 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10e1c51dda675c0a786c24efd613f4540a95953278a436468ff426c7ae51d938\": container with ID starting with 10e1c51dda675c0a786c24efd613f4540a95953278a436468ff426c7ae51d938 not found: ID does not exist" containerID="10e1c51dda675c0a786c24efd613f4540a95953278a436468ff426c7ae51d938" Oct 30 07:18:03.437250 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:03.437207 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e1c51dda675c0a786c24efd613f4540a95953278a436468ff426c7ae51d938"} err="failed to get container status \"10e1c51dda675c0a786c24efd613f4540a95953278a436468ff426c7ae51d938\": rpc error: code = NotFound desc = could not find container \"10e1c51dda675c0a786c24efd613f4540a95953278a436468ff426c7ae51d938\": container with ID starting with 10e1c51dda675c0a786c24efd613f4540a95953278a436468ff426c7ae51d938 not found: ID does not exist" Oct 30 07:18:03.461641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:03.461617 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1e3dd1-695f9f69f9-w54sq"] Oct 30 07:18:03.471554 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:03.471531 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1e3dd1-695f9f69f9-w54sq"] Oct 30 07:18:03.500841 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:03.500815 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1e3dd1-789c944fd4-m74hx"] Oct 30 07:18:03.504390 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:03.504370 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1e3dd1-789c944fd4-m74hx"] Oct 30 07:18:03.773476 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:03.773444 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2d13e7a-5d8a-417f-aff7-746e41ba4510" path="/var/lib/kubelet/pods/d2d13e7a-5d8a-417f-aff7-746e41ba4510/volumes" Oct 30 07:18:03.774027 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:03.774008 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc1e4717-80e7-47d7-b7f7-75a9a2477d50" path="/var/lib/kubelet/pods/fc1e4717-80e7-47d7-b7f7-75a9a2477d50/volumes" Oct 30 07:18:07.506146 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:07.506105 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz"] Oct 30 07:18:07.507311 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:07.507283 4951 cpu_manager.go:414] "containerMap: removing container" podUID="d2d13e7a-5d8a-417f-aff7-746e41ba4510" containerName="payload-pre-processing" Oct 30 07:18:07.507311 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:07.507307 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2d13e7a-5d8a-417f-aff7-746e41ba4510" containerName="payload-pre-processing" Oct 30 07:18:07.507524 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:07.507329 4951 cpu_manager.go:414] "containerMap: removing container" podUID="fc1e4717-80e7-47d7-b7f7-75a9a2477d50" containerName="payload-processing" Oct 30 07:18:07.507524 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:07.507334 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc1e4717-80e7-47d7-b7f7-75a9a2477d50" containerName="payload-processing" Oct 30 07:18:07.507524 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:07.507370 4951 cpu_manager.go:414] "containerMap: removing container" podUID="1e764719-1b26-4ba1-87ba-fe5499e58071" containerName="payload-processing" Oct 30 07:18:07.507524 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:07.507376 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e764719-1b26-4ba1-87ba-fe5499e58071" containerName="payload-processing" Oct 30 07:18:07.507524 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:07.507504 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2d13e7a-5d8a-417f-aff7-746e41ba4510" containerName="payload-pre-processing" Oct 30 07:18:07.507524 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:07.507517 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc1e4717-80e7-47d7-b7f7-75a9a2477d50" containerName="payload-processing" Oct 30 07:18:07.507524 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:07.507527 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e764719-1b26-4ba1-87ba-fe5499e58071" containerName="payload-processing" Oct 30 07:18:07.511547 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:07.511519 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz" Oct 30 07:18:07.531474 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:07.531450 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-4c43f7-serving-cert\"" Oct 30 07:18:07.537177 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:07.537142 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz"] Oct 30 07:18:07.556265 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:07.556223 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2076f66-8e90-4c87-ab08-6d4b5e3280b6-openshift-service-ca\") pod \"maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz\" (UID: \"b2076f66-8e90-4c87-ab08-6d4b5e3280b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz" Oct 30 07:18:07.556681 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:07.556644 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwwd9\" (UniqueName: \"kubernetes.io/projected/b2076f66-8e90-4c87-ab08-6d4b5e3280b6-kube-api-access-gwwd9\") pod \"maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz\" (UID: \"b2076f66-8e90-4c87-ab08-6d4b5e3280b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz" Oct 30 07:18:07.556794 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:07.556743 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2076f66-8e90-4c87-ab08-6d4b5e3280b6-maas-api-tls\") pod \"maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz\" (UID: \"b2076f66-8e90-4c87-ab08-6d4b5e3280b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz" Oct 30 07:18:07.556850 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:07.556824 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2076f66-8e90-4c87-ab08-6d4b5e3280b6-metrics-certs\") pod \"maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz\" (UID: \"b2076f66-8e90-4c87-ab08-6d4b5e3280b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz" Oct 30 07:18:07.658390 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:07.658353 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwwd9\" (UniqueName: \"kubernetes.io/projected/b2076f66-8e90-4c87-ab08-6d4b5e3280b6-kube-api-access-gwwd9\") pod \"maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz\" (UID: \"b2076f66-8e90-4c87-ab08-6d4b5e3280b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz" Oct 30 07:18:07.658628 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:07.658457 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2076f66-8e90-4c87-ab08-6d4b5e3280b6-maas-api-tls\") pod \"maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz\" (UID: \"b2076f66-8e90-4c87-ab08-6d4b5e3280b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz" Oct 30 07:18:07.658628 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:07.658532 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2076f66-8e90-4c87-ab08-6d4b5e3280b6-metrics-certs\") pod \"maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz\" (UID: \"b2076f66-8e90-4c87-ab08-6d4b5e3280b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz" Oct 30 07:18:07.658798 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:07.658780 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2076f66-8e90-4c87-ab08-6d4b5e3280b6-openshift-service-ca\") pod \"maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz\" (UID: \"b2076f66-8e90-4c87-ab08-6d4b5e3280b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz" Oct 30 07:18:07.659431 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:07.659404 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2076f66-8e90-4c87-ab08-6d4b5e3280b6-openshift-service-ca\") pod \"maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz\" (UID: \"b2076f66-8e90-4c87-ab08-6d4b5e3280b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz" Oct 30 07:18:07.661162 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:07.661138 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2076f66-8e90-4c87-ab08-6d4b5e3280b6-metrics-certs\") pod \"maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz\" (UID: \"b2076f66-8e90-4c87-ab08-6d4b5e3280b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz" Oct 30 07:18:07.661278 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:07.661185 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2076f66-8e90-4c87-ab08-6d4b5e3280b6-maas-api-tls\") pod \"maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz\" (UID: \"b2076f66-8e90-4c87-ab08-6d4b5e3280b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz" Oct 30 07:18:07.679597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:07.679543 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwwd9\" (UniqueName: \"kubernetes.io/projected/b2076f66-8e90-4c87-ab08-6d4b5e3280b6-kube-api-access-gwwd9\") pod \"maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz\" (UID: \"b2076f66-8e90-4c87-ab08-6d4b5e3280b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz" Oct 30 07:18:07.837446 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:07.835897 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz" Oct 30 07:18:08.208211 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:08.208180 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz"] Oct 30 07:18:08.209476 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:18:08.209448 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2076f66_8e90_4c87_ab08_6d4b5e3280b6.slice/crio-8374e74ad159eca3a94d9dea4867e9b76566cf4fa21e5ac0acb685dec6b3a503 WatchSource:0}: Error finding container 8374e74ad159eca3a94d9dea4867e9b76566cf4fa21e5ac0acb685dec6b3a503: Status 404 returned error can't find the container with id 8374e74ad159eca3a94d9dea4867e9b76566cf4fa21e5ac0acb685dec6b3a503 Oct 30 07:18:08.554680 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:08.554643 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz" event={"ID":"b2076f66-8e90-4c87-ab08-6d4b5e3280b6","Type":"ContainerStarted","Data":"8374e74ad159eca3a94d9dea4867e9b76566cf4fa21e5ac0acb685dec6b3a503"} Oct 30 07:18:09.565799 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:09.565744 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz" event={"ID":"b2076f66-8e90-4c87-ab08-6d4b5e3280b6","Type":"ContainerStarted","Data":"d0730eb23d2e0d6d54ff18e863fc16a0109b32f0bcecac60e5d38054771ca55e"} Oct 30 07:18:09.566272 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:09.565825 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz" Oct 30 07:18:09.610238 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:09.610166 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz" podStartSLOduration=2.247716809 podStartE2EDuration="2.610149449s" totalImagesPullingTime="362.43264ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:18:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:18:09.603924727 +0000 UTC m=+1295.946972693" watchObservedRunningTime="2026-10-30 07:18:09.610149449 +0000 UTC m=+1295.953197416" Oct 30 07:18:10.372122 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:10.372092 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-486cd4-5758d6dfc9-v5nfj" Oct 30 07:18:10.372503 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:10.372484 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-486cd4-5db6fd58-tg4wr" Oct 30 07:18:13.398848 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:13.398813 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-486cd4-79d7766689-przsr"] Oct 30 07:18:13.403235 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:13.403213 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-486cd4-79d7766689-przsr" Oct 30 07:18:13.433490 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:13.433454 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-486cd4-serving-cert\"" Oct 30 07:18:13.440330 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:13.440290 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-486cd4-79d7766689-przsr"] Oct 30 07:18:13.506485 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:13.506445 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzpk7\" (UniqueName: \"kubernetes.io/projected/e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4-kube-api-access-nzpk7\") pod \"maas-api-e2e-api-b-486cd4-79d7766689-przsr\" (UID: \"e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-486cd4-79d7766689-przsr" Oct 30 07:18:13.506694 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:13.506653 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4-metrics-certs\") pod \"maas-api-e2e-api-b-486cd4-79d7766689-przsr\" (UID: \"e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-486cd4-79d7766689-przsr" Oct 30 07:18:13.506807 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:13.506790 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4-openshift-service-ca\") pod \"maas-api-e2e-api-b-486cd4-79d7766689-przsr\" (UID: \"e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-486cd4-79d7766689-przsr" Oct 30 07:18:13.506858 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:13.506847 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4-maas-api-tls\") pod \"maas-api-e2e-api-b-486cd4-79d7766689-przsr\" (UID: \"e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-486cd4-79d7766689-przsr" Oct 30 07:18:13.608786 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:13.608748 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzpk7\" (UniqueName: \"kubernetes.io/projected/e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4-kube-api-access-nzpk7\") pod \"maas-api-e2e-api-b-486cd4-79d7766689-przsr\" (UID: \"e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-486cd4-79d7766689-przsr" Oct 30 07:18:13.609208 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:13.609170 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4-metrics-certs\") pod \"maas-api-e2e-api-b-486cd4-79d7766689-przsr\" (UID: \"e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-486cd4-79d7766689-przsr" Oct 30 07:18:13.609461 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:13.609444 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4-openshift-service-ca\") pod \"maas-api-e2e-api-b-486cd4-79d7766689-przsr\" (UID: \"e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-486cd4-79d7766689-przsr" Oct 30 07:18:13.609616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:13.609601 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4-maas-api-tls\") pod \"maas-api-e2e-api-b-486cd4-79d7766689-przsr\" (UID: \"e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-486cd4-79d7766689-przsr" Oct 30 07:18:13.611412 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:13.611385 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4-openshift-service-ca\") pod \"maas-api-e2e-api-b-486cd4-79d7766689-przsr\" (UID: \"e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-486cd4-79d7766689-przsr" Oct 30 07:18:13.642295 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:13.642211 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4-metrics-certs\") pod \"maas-api-e2e-api-b-486cd4-79d7766689-przsr\" (UID: \"e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-486cd4-79d7766689-przsr" Oct 30 07:18:13.652811 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:13.652657 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4-maas-api-tls\") pod \"maas-api-e2e-api-b-486cd4-79d7766689-przsr\" (UID: \"e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-486cd4-79d7766689-przsr" Oct 30 07:18:13.653875 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:13.653811 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzpk7\" (UniqueName: \"kubernetes.io/projected/e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4-kube-api-access-nzpk7\") pod \"maas-api-e2e-api-b-486cd4-79d7766689-przsr\" (UID: \"e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-486cd4-79d7766689-przsr" Oct 30 07:18:13.738909 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:13.738860 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-486cd4-79d7766689-przsr" Oct 30 07:18:14.090236 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:14.090202 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-486cd4-79d7766689-przsr"] Oct 30 07:18:14.090401 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:18:14.090370 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode67ce57d_fbc0_49f6_b16c_fa7a5256a6e4.slice/crio-31849c59d900615d8f314457bd7f37f98dbf42bebe0680401d3c1dabe784c31f WatchSource:0}: Error finding container 31849c59d900615d8f314457bd7f37f98dbf42bebe0680401d3c1dabe784c31f: Status 404 returned error can't find the container with id 31849c59d900615d8f314457bd7f37f98dbf42bebe0680401d3c1dabe784c31f Oct 30 07:18:14.766065 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:14.766020 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-486cd4-79d7766689-przsr" event={"ID":"e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4","Type":"ContainerStarted","Data":"b08ea5bc595cbaae109f1a370dff033347034e547ee333e1f28ef3248564ba13"} Oct 30 07:18:14.766664 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:14.766637 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-486cd4-79d7766689-przsr" event={"ID":"e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4","Type":"ContainerStarted","Data":"31849c59d900615d8f314457bd7f37f98dbf42bebe0680401d3c1dabe784c31f"} Oct 30 07:18:14.768005 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:14.767969 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-486cd4-79d7766689-przsr" Oct 30 07:18:14.825019 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:14.821054 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-486cd4-79d7766689-przsr" podStartSLOduration=1.490885525 podStartE2EDuration="1.821027794s" totalImagesPullingTime="330.142269ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:18:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:18:14.818040515 +0000 UTC m=+1301.161088481" watchObservedRunningTime="2026-10-30 07:18:14.821027794 +0000 UTC m=+1301.164075763" Oct 30 07:18:15.591747 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:15.591663 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz" Oct 30 07:18:20.146710 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:20.146666 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd"] Oct 30 07:18:20.147194 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:20.146945 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd" podUID="369630cd-08d5-4b99-9a86-c33e774becf7" containerName="maas-api" containerID="cri-o://ea1fa2ba279379b5c03a5de510bfa905bbe16fa41005e98475058febce10cbb4" gracePeriod=30 Oct 30 07:18:20.826649 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:20.808907 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd" Oct 30 07:18:20.858593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:20.852983 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/369630cd-08d5-4b99-9a86-c33e774becf7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/369630cd-08d5-4b99-9a86-c33e774becf7-metrics-certs\") pod \"369630cd-08d5-4b99-9a86-c33e774becf7\" (UID: \"369630cd-08d5-4b99-9a86-c33e774becf7\") " Oct 30 07:18:20.858593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:20.853134 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/369630cd-08d5-4b99-9a86-c33e774becf7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/369630cd-08d5-4b99-9a86-c33e774becf7-openshift-service-ca\") pod \"369630cd-08d5-4b99-9a86-c33e774becf7\" (UID: \"369630cd-08d5-4b99-9a86-c33e774becf7\") " Oct 30 07:18:20.858593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:20.853240 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/369630cd-08d5-4b99-9a86-c33e774becf7-kube-api-access-tlbmj\" (UniqueName: \"kubernetes.io/projected/369630cd-08d5-4b99-9a86-c33e774becf7-kube-api-access-tlbmj\") pod \"369630cd-08d5-4b99-9a86-c33e774becf7\" (UID: \"369630cd-08d5-4b99-9a86-c33e774becf7\") " Oct 30 07:18:20.858593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:20.853522 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/369630cd-08d5-4b99-9a86-c33e774becf7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/369630cd-08d5-4b99-9a86-c33e774becf7-maas-api-tls\") pod \"369630cd-08d5-4b99-9a86-c33e774becf7\" (UID: \"369630cd-08d5-4b99-9a86-c33e774becf7\") " Oct 30 07:18:20.904088 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:20.859125 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/369630cd-08d5-4b99-9a86-c33e774becf7-maas-api-tls" pod "369630cd-08d5-4b99-9a86-c33e774becf7" (UID: "369630cd-08d5-4b99-9a86-c33e774becf7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:18:20.904088 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:20.859612 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/369630cd-08d5-4b99-9a86-c33e774becf7-openshift-service-ca" pod "369630cd-08d5-4b99-9a86-c33e774becf7" (UID: "369630cd-08d5-4b99-9a86-c33e774becf7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:18:20.904088 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:20.894338 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/369630cd-08d5-4b99-9a86-c33e774becf7-metrics-certs" pod "369630cd-08d5-4b99-9a86-c33e774becf7" (UID: "369630cd-08d5-4b99-9a86-c33e774becf7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:18:20.958650 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:20.958261 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/369630cd-08d5-4b99-9a86-c33e774becf7-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:20.958650 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:20.958299 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/369630cd-08d5-4b99-9a86-c33e774becf7-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:20.958650 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:20.958316 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/369630cd-08d5-4b99-9a86-c33e774becf7-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:20.987335 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:20.978407 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/369630cd-08d5-4b99-9a86-c33e774becf7-kube-api-access-tlbmj" pod "369630cd-08d5-4b99-9a86-c33e774becf7" (UID: "369630cd-08d5-4b99-9a86-c33e774becf7"). InnerVolumeSpecName "kube-api-access-tlbmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:21.036084 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:21.035221 4951 generic.go:356] "Generic (PLEG): container finished" podID="369630cd-08d5-4b99-9a86-c33e774becf7" containerID="ea1fa2ba279379b5c03a5de510bfa905bbe16fa41005e98475058febce10cbb4" exitCode=0 Oct 30 07:18:21.036084 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:21.035270 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd" Oct 30 07:18:21.036084 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:21.035455 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd" event={"ID":"369630cd-08d5-4b99-9a86-c33e774becf7","Type":"ContainerDied","Data":"ea1fa2ba279379b5c03a5de510bfa905bbe16fa41005e98475058febce10cbb4"} Oct 30 07:18:21.036084 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:21.035496 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd" event={"ID":"369630cd-08d5-4b99-9a86-c33e774becf7","Type":"ContainerDied","Data":"f02265ecc8e8816e92d49f60471aa94011bf7f188fb760ac2ed1add4f10c5406"} Oct 30 07:18:21.036084 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:21.035524 4951 scope.go:122] "RemoveContainer" containerID="ea1fa2ba279379b5c03a5de510bfa905bbe16fa41005e98475058febce10cbb4" Oct 30 07:18:21.063619 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:21.062905 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlbmj\" (UniqueName: \"kubernetes.io/projected/369630cd-08d5-4b99-9a86-c33e774becf7-kube-api-access-tlbmj\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:21.146994 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:21.145773 4951 scope.go:122] "RemoveContainer" containerID="ea1fa2ba279379b5c03a5de510bfa905bbe16fa41005e98475058febce10cbb4" Oct 30 07:18:21.189631 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:18:21.181970 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea1fa2ba279379b5c03a5de510bfa905bbe16fa41005e98475058febce10cbb4\": container with ID starting with ea1fa2ba279379b5c03a5de510bfa905bbe16fa41005e98475058febce10cbb4 not found: ID does not exist" containerID="ea1fa2ba279379b5c03a5de510bfa905bbe16fa41005e98475058febce10cbb4" Oct 30 07:18:21.189631 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:21.182012 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea1fa2ba279379b5c03a5de510bfa905bbe16fa41005e98475058febce10cbb4"} err="failed to get container status \"ea1fa2ba279379b5c03a5de510bfa905bbe16fa41005e98475058febce10cbb4\": rpc error: code = NotFound desc = could not find container \"ea1fa2ba279379b5c03a5de510bfa905bbe16fa41005e98475058febce10cbb4\": container with ID starting with ea1fa2ba279379b5c03a5de510bfa905bbe16fa41005e98475058febce10cbb4 not found: ID does not exist" Oct 30 07:18:21.198254 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:21.197044 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd"] Oct 30 07:18:21.207618 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:21.207545 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1e3dd1-696656799b-8gbkd"] Oct 30 07:18:21.777437 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:21.777404 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="369630cd-08d5-4b99-9a86-c33e774becf7" path="/var/lib/kubelet/pods/369630cd-08d5-4b99-9a86-c33e774becf7/volumes" Oct 30 07:18:21.786421 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:21.786394 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-486cd4-79d7766689-przsr" Oct 30 07:18:23.025838 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:18:23.025796 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:18:29.193203 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:29.193169 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs"] Oct 30 07:18:29.193802 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:29.193421 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs" podUID="fabd32c8-62a7-44ac-9708-e5b56f695b12" containerName="maas-api" containerID="cri-o://da119fb51dbbb6eb9045c67ee941797015823e331413311c0a5d818d35a05c2f" gracePeriod=30 Oct 30 07:18:29.727596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:29.723824 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs" Oct 30 07:18:29.852650 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:29.852359 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fabd32c8-62a7-44ac-9708-e5b56f695b12-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fabd32c8-62a7-44ac-9708-e5b56f695b12-metrics-certs\") pod \"fabd32c8-62a7-44ac-9708-e5b56f695b12\" (UID: \"fabd32c8-62a7-44ac-9708-e5b56f695b12\") " Oct 30 07:18:29.852650 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:29.852423 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fabd32c8-62a7-44ac-9708-e5b56f695b12-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fabd32c8-62a7-44ac-9708-e5b56f695b12-openshift-service-ca\") pod \"fabd32c8-62a7-44ac-9708-e5b56f695b12\" (UID: \"fabd32c8-62a7-44ac-9708-e5b56f695b12\") " Oct 30 07:18:29.868888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:29.853678 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fabd32c8-62a7-44ac-9708-e5b56f695b12-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fabd32c8-62a7-44ac-9708-e5b56f695b12-maas-api-tls\") pod \"fabd32c8-62a7-44ac-9708-e5b56f695b12\" (UID: \"fabd32c8-62a7-44ac-9708-e5b56f695b12\") " Oct 30 07:18:29.868888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:29.853722 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fabd32c8-62a7-44ac-9708-e5b56f695b12-kube-api-access-5wk6d\" (UniqueName: \"kubernetes.io/projected/fabd32c8-62a7-44ac-9708-e5b56f695b12-kube-api-access-5wk6d\") pod \"fabd32c8-62a7-44ac-9708-e5b56f695b12\" (UID: \"fabd32c8-62a7-44ac-9708-e5b56f695b12\") " Oct 30 07:18:29.868888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:29.861946 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fabd32c8-62a7-44ac-9708-e5b56f695b12-openshift-service-ca" pod "fabd32c8-62a7-44ac-9708-e5b56f695b12" (UID: "fabd32c8-62a7-44ac-9708-e5b56f695b12"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:18:29.901052 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:29.888087 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fabd32c8-62a7-44ac-9708-e5b56f695b12-metrics-certs" pod "fabd32c8-62a7-44ac-9708-e5b56f695b12" (UID: "fabd32c8-62a7-44ac-9708-e5b56f695b12"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:18:29.937602 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:29.927178 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fabd32c8-62a7-44ac-9708-e5b56f695b12-kube-api-access-5wk6d" pod "fabd32c8-62a7-44ac-9708-e5b56f695b12" (UID: "fabd32c8-62a7-44ac-9708-e5b56f695b12"). InnerVolumeSpecName "kube-api-access-5wk6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:30.019595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:30.018738 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wk6d\" (UniqueName: \"kubernetes.io/projected/fabd32c8-62a7-44ac-9708-e5b56f695b12-kube-api-access-5wk6d\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:30.019595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:30.018779 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fabd32c8-62a7-44ac-9708-e5b56f695b12-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:30.019595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:30.018796 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fabd32c8-62a7-44ac-9708-e5b56f695b12-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:30.043240 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:30.043191 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fabd32c8-62a7-44ac-9708-e5b56f695b12-maas-api-tls" pod "fabd32c8-62a7-44ac-9708-e5b56f695b12" (UID: "fabd32c8-62a7-44ac-9708-e5b56f695b12"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:18:30.135678 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:30.135381 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fabd32c8-62a7-44ac-9708-e5b56f695b12-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:30.201387 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:30.201353 4951 generic.go:356] "Generic (PLEG): container finished" podID="fabd32c8-62a7-44ac-9708-e5b56f695b12" containerID="da119fb51dbbb6eb9045c67ee941797015823e331413311c0a5d818d35a05c2f" exitCode=0 Oct 30 07:18:30.202069 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:30.202040 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs" event={"ID":"fabd32c8-62a7-44ac-9708-e5b56f695b12","Type":"ContainerDied","Data":"da119fb51dbbb6eb9045c67ee941797015823e331413311c0a5d818d35a05c2f"} Oct 30 07:18:30.202245 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:30.202230 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs" event={"ID":"fabd32c8-62a7-44ac-9708-e5b56f695b12","Type":"ContainerDied","Data":"1447116c78ed59e9d5d5e4e31672e83deab21b2a1da71c71a58f560d6b096abc"} Oct 30 07:18:30.202367 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:30.202353 4951 scope.go:122] "RemoveContainer" containerID="da119fb51dbbb6eb9045c67ee941797015823e331413311c0a5d818d35a05c2f" Oct 30 07:18:30.202643 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:30.202627 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs" Oct 30 07:18:30.263609 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:30.263382 4951 scope.go:122] "RemoveContainer" containerID="da119fb51dbbb6eb9045c67ee941797015823e331413311c0a5d818d35a05c2f" Oct 30 07:18:30.266618 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:18:30.263909 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da119fb51dbbb6eb9045c67ee941797015823e331413311c0a5d818d35a05c2f\": container with ID starting with da119fb51dbbb6eb9045c67ee941797015823e331413311c0a5d818d35a05c2f not found: ID does not exist" containerID="da119fb51dbbb6eb9045c67ee941797015823e331413311c0a5d818d35a05c2f" Oct 30 07:18:30.266618 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:30.263946 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da119fb51dbbb6eb9045c67ee941797015823e331413311c0a5d818d35a05c2f"} err="failed to get container status \"da119fb51dbbb6eb9045c67ee941797015823e331413311c0a5d818d35a05c2f\": rpc error: code = NotFound desc = could not find container \"da119fb51dbbb6eb9045c67ee941797015823e331413311c0a5d818d35a05c2f\": container with ID starting with da119fb51dbbb6eb9045c67ee941797015823e331413311c0a5d818d35a05c2f not found: ID does not exist" Oct 30 07:18:30.291593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:30.291347 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs"] Oct 30 07:18:30.305839 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:30.305612 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-56eb7f-54d8c66b9f-zz7vs"] Oct 30 07:18:31.303385 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.303345 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2"] Oct 30 07:18:31.304671 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.304643 4951 cpu_manager.go:414] "containerMap: removing container" podUID="fabd32c8-62a7-44ac-9708-e5b56f695b12" containerName="maas-api" Oct 30 07:18:31.304671 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.304667 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fabd32c8-62a7-44ac-9708-e5b56f695b12" containerName="maas-api" Oct 30 07:18:31.304878 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.304704 4951 cpu_manager.go:414] "containerMap: removing container" podUID="369630cd-08d5-4b99-9a86-c33e774becf7" containerName="maas-api" Oct 30 07:18:31.304878 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.304709 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="369630cd-08d5-4b99-9a86-c33e774becf7" containerName="maas-api" Oct 30 07:18:31.304985 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.304878 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="fabd32c8-62a7-44ac-9708-e5b56f695b12" containerName="maas-api" Oct 30 07:18:31.304985 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.304906 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="369630cd-08d5-4b99-9a86-c33e774becf7" containerName="maas-api" Oct 30 07:18:31.310044 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.310016 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:18:31.383105 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.342074 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-052254-openshift-default-dockercfg-xtqh5\"" Oct 30 07:18:31.383563 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.383540 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2"] Oct 30 07:18:31.493363 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.493315 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-credential-socket\") pod \"e2e-api-b-052254-openshift-default-57b897c995-9hsd2\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:18:31.493716 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.493691 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-workload-certs\") pod \"e2e-api-b-052254-openshift-default-57b897c995-9hsd2\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:18:31.493975 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.493955 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-workload-socket\") pod \"e2e-api-b-052254-openshift-default-57b897c995-9hsd2\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:18:31.494102 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.494087 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94280cc1-4ffe-4862-9cfc-1d61949679ed-istiod-ca-cert\") pod \"e2e-api-b-052254-openshift-default-57b897c995-9hsd2\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:18:31.515598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.494190 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdn8d\" (UniqueName: \"kubernetes.io/projected/94280cc1-4ffe-4862-9cfc-1d61949679ed-kube-api-access-hdn8d\") pod \"e2e-api-b-052254-openshift-default-57b897c995-9hsd2\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:18:31.515915 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.515878 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94280cc1-4ffe-4862-9cfc-1d61949679ed-istio-token\") pod \"e2e-api-b-052254-openshift-default-57b897c995-9hsd2\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:18:31.516142 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.516123 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-istio-data\") pod \"e2e-api-b-052254-openshift-default-57b897c995-9hsd2\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:18:31.516293 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.516278 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-istio-envoy\") pod \"e2e-api-b-052254-openshift-default-57b897c995-9hsd2\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:18:31.516479 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.516462 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94280cc1-4ffe-4862-9cfc-1d61949679ed-istio-podinfo\") pod \"e2e-api-b-052254-openshift-default-57b897c995-9hsd2\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:18:31.640645 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.640169 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-workload-socket\") pod \"e2e-api-b-052254-openshift-default-57b897c995-9hsd2\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:18:31.640645 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.640231 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94280cc1-4ffe-4862-9cfc-1d61949679ed-istiod-ca-cert\") pod \"e2e-api-b-052254-openshift-default-57b897c995-9hsd2\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:18:31.640645 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.640263 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdn8d\" (UniqueName: \"kubernetes.io/projected/94280cc1-4ffe-4862-9cfc-1d61949679ed-kube-api-access-hdn8d\") pod \"e2e-api-b-052254-openshift-default-57b897c995-9hsd2\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:18:31.640645 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.640297 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94280cc1-4ffe-4862-9cfc-1d61949679ed-istio-token\") pod \"e2e-api-b-052254-openshift-default-57b897c995-9hsd2\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:18:31.640645 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.640344 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-istio-data\") pod \"e2e-api-b-052254-openshift-default-57b897c995-9hsd2\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:18:31.640645 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.640392 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-istio-envoy\") pod \"e2e-api-b-052254-openshift-default-57b897c995-9hsd2\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:18:31.640645 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.640472 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94280cc1-4ffe-4862-9cfc-1d61949679ed-istio-podinfo\") pod \"e2e-api-b-052254-openshift-default-57b897c995-9hsd2\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:18:31.641148 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.640664 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-credential-socket\") pod \"e2e-api-b-052254-openshift-default-57b897c995-9hsd2\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:18:31.641148 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.640777 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-workload-certs\") pod \"e2e-api-b-052254-openshift-default-57b897c995-9hsd2\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:18:31.672599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.641321 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-workload-certs\") pod \"e2e-api-b-052254-openshift-default-57b897c995-9hsd2\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:18:31.672599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.653974 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-workload-socket\") pod \"e2e-api-b-052254-openshift-default-57b897c995-9hsd2\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:18:31.672599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.655090 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94280cc1-4ffe-4862-9cfc-1d61949679ed-istiod-ca-cert\") pod \"e2e-api-b-052254-openshift-default-57b897c995-9hsd2\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:18:31.672599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.655948 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-istio-data\") pod \"e2e-api-b-052254-openshift-default-57b897c995-9hsd2\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:18:31.672599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.660921 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-credential-socket\") pod \"e2e-api-b-052254-openshift-default-57b897c995-9hsd2\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:18:31.700596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.686470 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-istio-envoy\") pod \"e2e-api-b-052254-openshift-default-57b897c995-9hsd2\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:18:31.700596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.687631 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94280cc1-4ffe-4862-9cfc-1d61949679ed-istio-podinfo\") pod \"e2e-api-b-052254-openshift-default-57b897c995-9hsd2\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:18:31.700596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.697435 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94280cc1-4ffe-4862-9cfc-1d61949679ed-istio-token\") pod \"e2e-api-b-052254-openshift-default-57b897c995-9hsd2\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:18:31.724597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.723917 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdn8d\" (UniqueName: \"kubernetes.io/projected/94280cc1-4ffe-4862-9cfc-1d61949679ed-kube-api-access-hdn8d\") pod \"e2e-api-b-052254-openshift-default-57b897c995-9hsd2\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:18:31.903327 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.903239 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fabd32c8-62a7-44ac-9708-e5b56f695b12" path="/var/lib/kubelet/pods/fabd32c8-62a7-44ac-9708-e5b56f695b12/volumes" Oct 30 07:18:31.993603 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:31.981918 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:18:32.510489 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:32.510225 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2"] Oct 30 07:18:32.511970 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:18:32.511939 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94280cc1_4ffe_4862_9cfc_1d61949679ed.slice/crio-37199ba5973ffbc8981500527ddd983e2cc49118cfa8bd242e6513ac43966396 WatchSource:0}: Error finding container 37199ba5973ffbc8981500527ddd983e2cc49118cfa8bd242e6513ac43966396: Status 404 returned error can't find the container with id 37199ba5973ffbc8981500527ddd983e2cc49118cfa8bd242e6513ac43966396 Oct 30 07:18:32.514660 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:32.514623 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:18:32.514766 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:32.514692 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:18:32.514766 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:32.514724 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:18:33.250546 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:33.250505 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" event={"ID":"94280cc1-4ffe-4862-9cfc-1d61949679ed","Type":"ContainerStarted","Data":"f10083df04ab335b726acd7c22e316dd8cabbf4aabd50511ecd7911f40f4c291"} Oct 30 07:18:33.250795 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:33.250585 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" event={"ID":"94280cc1-4ffe-4862-9cfc-1d61949679ed","Type":"ContainerStarted","Data":"37199ba5973ffbc8981500527ddd983e2cc49118cfa8bd242e6513ac43966396"} Oct 30 07:18:33.315466 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:33.309617 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" podStartSLOduration=2.3095961369999998 podStartE2EDuration="2.309596137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:18:33.307434828 +0000 UTC m=+1319.650482816" watchObservedRunningTime="2026-10-30 07:18:33.309596137 +0000 UTC m=+1319.652644103" Oct 30 07:18:33.998706 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:33.996990 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:18:34.021902 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:34.021465 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:18:34.414424 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:34.414359 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:18:34.416175 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:34.415900 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:18:37.587712 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:37.587672 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-052254-589746f98c-l74cl"] Oct 30 07:18:37.595144 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:37.595113 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-052254-589746f98c-l74cl" Oct 30 07:18:37.605736 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:37.605704 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-052254-589746f98c-l74cl"] Oct 30 07:18:37.668599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:37.667978 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-052254-dockercfg-98x75\"" Oct 30 07:18:37.668599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:37.668082 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-052254\"" Oct 30 07:18:37.985488 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:37.983610 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79ld5\" (UniqueName: \"kubernetes.io/projected/96900a43-4be1-46f0-ab3f-9162fa89f534-kube-api-access-79ld5\") pod \"payload-pre-processing-e2e-api-b-052254-589746f98c-l74cl\" (UID: \"96900a43-4be1-46f0-ab3f-9162fa89f534\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-052254-589746f98c-l74cl" Oct 30 07:18:37.985488 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:37.983688 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96900a43-4be1-46f0-ab3f-9162fa89f534-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-052254-589746f98c-l74cl\" (UID: \"96900a43-4be1-46f0-ab3f-9162fa89f534\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-052254-589746f98c-l74cl" Oct 30 07:18:38.089595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:38.088177 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79ld5\" (UniqueName: \"kubernetes.io/projected/96900a43-4be1-46f0-ab3f-9162fa89f534-kube-api-access-79ld5\") pod \"payload-pre-processing-e2e-api-b-052254-589746f98c-l74cl\" (UID: \"96900a43-4be1-46f0-ab3f-9162fa89f534\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-052254-589746f98c-l74cl" Oct 30 07:18:38.089595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:38.088298 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96900a43-4be1-46f0-ab3f-9162fa89f534-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-052254-589746f98c-l74cl\" (UID: \"96900a43-4be1-46f0-ab3f-9162fa89f534\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-052254-589746f98c-l74cl" Oct 30 07:18:38.089595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:38.089448 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96900a43-4be1-46f0-ab3f-9162fa89f534-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-052254-589746f98c-l74cl\" (UID: \"96900a43-4be1-46f0-ab3f-9162fa89f534\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-052254-589746f98c-l74cl" Oct 30 07:18:38.147311 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:38.147242 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79ld5\" (UniqueName: \"kubernetes.io/projected/96900a43-4be1-46f0-ab3f-9162fa89f534-kube-api-access-79ld5\") pod \"payload-pre-processing-e2e-api-b-052254-589746f98c-l74cl\" (UID: \"96900a43-4be1-46f0-ab3f-9162fa89f534\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-052254-589746f98c-l74cl" Oct 30 07:18:38.232495 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:38.219875 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-052254-589746f98c-l74cl" Oct 30 07:18:38.446931 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:38.446478 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td"] Oct 30 07:18:38.446931 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:38.446831 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" podUID="1007f3bc-edf3-41a8-b5cf-00bac3178407" containerName="istio-proxy" containerID="cri-o://728ed508cb0b801bc2352dfd313ba08a587530d2a9b97b6994d6e333276b4c38" gracePeriod=30 Oct 30 07:18:38.456556 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:38.456278 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8"] Oct 30 07:18:38.456753 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:38.456603 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" podUID="012bfdea-4eb7-442d-a951-d1fd23882318" containerName="istio-proxy" containerID="cri-o://8c0a7b6f94a9b8c382354be1b87553e07f84fe070e21a8c795e58aef60fe5ac0" gracePeriod=30 Oct 30 07:18:38.477774 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:38.477617 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns"] Oct 30 07:18:38.477939 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:38.477892 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" podUID="7e1deed7-8d57-42b0-8277-96ad692ca910" containerName="istio-proxy" containerID="cri-o://961594a7e4af9c81a6ce86df5f263720e9bc9723f7a6f0ed61821d248038ce50" gracePeriod=30 Oct 30 07:18:38.496179 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:38.482642 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m"] Oct 30 07:18:38.496179 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:38.482886 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" podUID="4070bffc-3574-4ee1-b444-e405af814c82" containerName="istio-proxy" containerID="cri-o://ab77304f3a90305cc11fbe0ba43694d858a760ebe62c02b137ec514d418a6529" gracePeriod=30 Oct 30 07:18:38.501200 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:38.500324 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j"] Oct 30 07:18:38.501200 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:38.500590 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" podUID="c193cba6-1789-4935-835b-80afc22f58ad" containerName="istio-proxy" containerID="cri-o://f456dd4a2e31710d2c1418b6b27d835ebb38edcb7d51c8e26c04fbdae0af7f5c" gracePeriod=30 Oct 30 07:18:38.506765 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:38.502913 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9"] Oct 30 07:18:38.506765 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:38.503227 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" podUID="67817544-1eb4-4e44-ba6c-b84c63df5f76" containerName="istio-proxy" containerID="cri-o://d13f424ae56f9715ca3ff73c2ed395644125fa23e08d9a093dfaf9bf0971b704" gracePeriod=30 Oct 30 07:18:38.530431 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:38.527079 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb"] Oct 30 07:18:38.530431 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:38.527302 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" podUID="745992af-9150-4336-bf13-2e0f087214a5" containerName="istio-proxy" containerID="cri-o://3343d0803013b4629c2128935e9174ec7feb1a7fefeede37759adb5a3284d980" gracePeriod=30 Oct 30 07:18:39.002913 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:39.002832 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-052254-589746f98c-l74cl"] Oct 30 07:18:39.042134 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:18:39.042095 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96900a43_4be1_46f0_ab3f_9162fa89f534.slice/crio-173add32c5d574f33dfa80c45623395e97297f5bb38395dfcc9bd59801036cec WatchSource:0}: Error finding container 173add32c5d574f33dfa80c45623395e97297f5bb38395dfcc9bd59801036cec: Status 404 returned error can't find the container with id 173add32c5d574f33dfa80c45623395e97297f5bb38395dfcc9bd59801036cec Oct 30 07:18:39.241829 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:39.241458 4951 patch_prober.go:29] interesting pod/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:18:39.241829 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:39.241519 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" podUID="f3d3f735-6cd0-4153-a6ff-88f77f4941ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:18:39.511151 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:39.511112 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-052254-589746f98c-l74cl" event={"ID":"96900a43-4be1-46f0-ab3f-9162fa89f534","Type":"ContainerStarted","Data":"9949ebc6fde83a21bea18d39a7901cb960a4e5d143cb6ac212e592372aa73b7e"} Oct 30 07:18:39.511352 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:39.511213 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-052254-589746f98c-l74cl" event={"ID":"96900a43-4be1-46f0-ab3f-9162fa89f534","Type":"ContainerStarted","Data":"173add32c5d574f33dfa80c45623395e97297f5bb38395dfcc9bd59801036cec"} Oct 30 07:18:39.511352 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:39.511245 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-052254-589746f98c-l74cl" Oct 30 07:18:39.557619 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:39.554979 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-052254-589746f98c-l74cl" podStartSLOduration=2.554961677 podStartE2EDuration="2.554961677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:18:39.550379688 +0000 UTC m=+1325.893427655" watchObservedRunningTime="2026-10-30 07:18:39.554961677 +0000 UTC m=+1325.898009654" Oct 30 07:18:42.539646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:42.539608 4951 generic.go:356] "Generic (PLEG): container finished" podID="f3d3f735-6cd0-4153-a6ff-88f77f4941ca" containerID="27a16ce17a1d15ea1f14c838de0bb4c00094047037482fb1b05dab8f0bbebc87" exitCode=0 Oct 30 07:18:42.540183 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:42.539663 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" event={"ID":"f3d3f735-6cd0-4153-a6ff-88f77f4941ca","Type":"ContainerDied","Data":"27a16ce17a1d15ea1f14c838de0bb4c00094047037482fb1b05dab8f0bbebc87"} Oct 30 07:18:42.540183 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:42.540113 4951 scope.go:122] "RemoveContainer" containerID="27a16ce17a1d15ea1f14c838de0bb4c00094047037482fb1b05dab8f0bbebc87" Oct 30 07:18:42.541775 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:42.541743 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:18:42.541908 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:42.541831 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:18:42.541908 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:42.541881 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:18:43.702989 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:43.702656 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" event={"ID":"f3d3f735-6cd0-4153-a6ff-88f77f4941ca","Type":"ContainerStarted","Data":"e7c7a4790b3a792a5a717ead1f3d71eaa2577132870247c020633cd6e3c619a2"} Oct 30 07:18:43.729716 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:43.728909 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:18:43.735973 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:43.735619 4951 patch_prober.go:29] interesting pod/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 07:18:43.735973 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:43.735671 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" podUID="f3d3f735-6cd0-4153-a6ff-88f77f4941ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 07:18:44.208726 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.207584 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:18:44.315918 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.307012 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-workload-socket\") pod \"67817544-1eb4-4e44-ba6c-b84c63df5f76\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " Oct 30 07:18:44.315918 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.307152 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-istio-data\") pod \"67817544-1eb4-4e44-ba6c-b84c63df5f76\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " Oct 30 07:18:44.315918 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.307188 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-istio-envoy\") pod \"67817544-1eb4-4e44-ba6c-b84c63df5f76\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " Oct 30 07:18:44.315918 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.307310 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67817544-1eb4-4e44-ba6c-b84c63df5f76-istio-token\" (UniqueName: \"kubernetes.io/projected/67817544-1eb4-4e44-ba6c-b84c63df5f76-istio-token\") pod \"67817544-1eb4-4e44-ba6c-b84c63df5f76\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " Oct 30 07:18:44.315918 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.307429 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67817544-1eb4-4e44-ba6c-b84c63df5f76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67817544-1eb4-4e44-ba6c-b84c63df5f76-istiod-ca-cert\") pod \"67817544-1eb4-4e44-ba6c-b84c63df5f76\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " Oct 30 07:18:44.315918 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.307467 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67817544-1eb4-4e44-ba6c-b84c63df5f76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67817544-1eb4-4e44-ba6c-b84c63df5f76-istio-podinfo\") pod \"67817544-1eb4-4e44-ba6c-b84c63df5f76\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " Oct 30 07:18:44.315918 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.307630 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-credential-socket\") pod \"67817544-1eb4-4e44-ba6c-b84c63df5f76\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " Oct 30 07:18:44.315918 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.307677 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67817544-1eb4-4e44-ba6c-b84c63df5f76-kube-api-access-mdwzx\" (UniqueName: \"kubernetes.io/projected/67817544-1eb4-4e44-ba6c-b84c63df5f76-kube-api-access-mdwzx\") pod \"67817544-1eb4-4e44-ba6c-b84c63df5f76\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " Oct 30 07:18:44.315918 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.307721 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-workload-certs\") pod \"67817544-1eb4-4e44-ba6c-b84c63df5f76\" (UID: \"67817544-1eb4-4e44-ba6c-b84c63df5f76\") " Oct 30 07:18:44.315918 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.308735 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-workload-certs" pod "67817544-1eb4-4e44-ba6c-b84c63df5f76" (UID: "67817544-1eb4-4e44-ba6c-b84c63df5f76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.315918 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.308962 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-workload-socket" pod "67817544-1eb4-4e44-ba6c-b84c63df5f76" (UID: "67817544-1eb4-4e44-ba6c-b84c63df5f76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.315918 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.309153 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-istio-data" pod "67817544-1eb4-4e44-ba6c-b84c63df5f76" (UID: "67817544-1eb4-4e44-ba6c-b84c63df5f76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.315918 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.309309 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67817544-1eb4-4e44-ba6c-b84c63df5f76-istiod-ca-cert" pod "67817544-1eb4-4e44-ba6c-b84c63df5f76" (UID: "67817544-1eb4-4e44-ba6c-b84c63df5f76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:18:44.315918 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.309895 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-credential-socket" pod "67817544-1eb4-4e44-ba6c-b84c63df5f76" (UID: "67817544-1eb4-4e44-ba6c-b84c63df5f76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.348722 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.344863 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67817544-1eb4-4e44-ba6c-b84c63df5f76-istio-token" pod "67817544-1eb4-4e44-ba6c-b84c63df5f76" (UID: "67817544-1eb4-4e44-ba6c-b84c63df5f76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:44.377017 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.376540 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67817544-1eb4-4e44-ba6c-b84c63df5f76-istio-podinfo" pod "67817544-1eb4-4e44-ba6c-b84c63df5f76" (UID: "67817544-1eb4-4e44-ba6c-b84c63df5f76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:18:44.377225 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.377035 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-istio-envoy" pod "67817544-1eb4-4e44-ba6c-b84c63df5f76" (UID: "67817544-1eb4-4e44-ba6c-b84c63df5f76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.381187 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.378410 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:18:44.381187 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.380534 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67817544-1eb4-4e44-ba6c-b84c63df5f76-kube-api-access-mdwzx" pod "67817544-1eb4-4e44-ba6c-b84c63df5f76" (UID: "67817544-1eb4-4e44-ba6c-b84c63df5f76"). InnerVolumeSpecName "kube-api-access-mdwzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:44.399459 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.397776 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:18:44.401680 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.401603 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:18:44.409877 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.409749 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:18:44.410694 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.410353 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e1deed7-8d57-42b0-8277-96ad692ca910-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e1deed7-8d57-42b0-8277-96ad692ca910-istio-podinfo\") pod \"7e1deed7-8d57-42b0-8277-96ad692ca910\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " Oct 30 07:18:44.410694 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.410400 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/012bfdea-4eb7-442d-a951-d1fd23882318-kube-api-access-f2wcm\" (UniqueName: \"kubernetes.io/projected/012bfdea-4eb7-442d-a951-d1fd23882318-kube-api-access-f2wcm\") pod \"012bfdea-4eb7-442d-a951-d1fd23882318\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " Oct 30 07:18:44.410694 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.410431 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e1deed7-8d57-42b0-8277-96ad692ca910-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e1deed7-8d57-42b0-8277-96ad692ca910-istiod-ca-cert\") pod \"7e1deed7-8d57-42b0-8277-96ad692ca910\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " Oct 30 07:18:44.410694 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.410484 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-istio-envoy\") pod \"7e1deed7-8d57-42b0-8277-96ad692ca910\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " Oct 30 07:18:44.410694 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.410540 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4070bffc-3574-4ee1-b444-e405af814c82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4070bffc-3574-4ee1-b444-e405af814c82-istio-podinfo\") pod \"4070bffc-3574-4ee1-b444-e405af814c82\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " Oct 30 07:18:44.411002 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.410780 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e1deed7-8d57-42b0-8277-96ad692ca910-istiod-ca-cert" pod "7e1deed7-8d57-42b0-8277-96ad692ca910" (UID: "7e1deed7-8d57-42b0-8277-96ad692ca910"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:18:44.412129 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.411109 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e1deed7-8d57-42b0-8277-96ad692ca910-istio-token\" (UniqueName: \"kubernetes.io/projected/7e1deed7-8d57-42b0-8277-96ad692ca910-istio-token\") pod \"7e1deed7-8d57-42b0-8277-96ad692ca910\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " Oct 30 07:18:44.412129 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.411166 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-workload-certs\") pod \"4070bffc-3574-4ee1-b444-e405af814c82\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " Oct 30 07:18:44.412129 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.411195 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-istio-envoy\") pod \"4070bffc-3574-4ee1-b444-e405af814c82\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " Oct 30 07:18:44.412129 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.411277 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-istio-data\") pod \"4070bffc-3574-4ee1-b444-e405af814c82\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " Oct 30 07:18:44.412129 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.411317 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e1deed7-8d57-42b0-8277-96ad692ca910-kube-api-access-6pwnt\" (UniqueName: \"kubernetes.io/projected/7e1deed7-8d57-42b0-8277-96ad692ca910-kube-api-access-6pwnt\") pod \"7e1deed7-8d57-42b0-8277-96ad692ca910\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " Oct 30 07:18:44.412129 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.411365 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-workload-socket\") pod \"7e1deed7-8d57-42b0-8277-96ad692ca910\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " Oct 30 07:18:44.412129 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.411406 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/012bfdea-4eb7-442d-a951-d1fd23882318-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012bfdea-4eb7-442d-a951-d1fd23882318-istio-podinfo\") pod \"012bfdea-4eb7-442d-a951-d1fd23882318\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " Oct 30 07:18:44.412129 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.411467 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-credential-socket\") pod \"7e1deed7-8d57-42b0-8277-96ad692ca910\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " Oct 30 07:18:44.412129 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.411504 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/012bfdea-4eb7-442d-a951-d1fd23882318-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012bfdea-4eb7-442d-a951-d1fd23882318-istiod-ca-cert\") pod \"012bfdea-4eb7-442d-a951-d1fd23882318\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " Oct 30 07:18:44.412129 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.411537 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4070bffc-3574-4ee1-b444-e405af814c82-kube-api-access-h5jpd\" (UniqueName: \"kubernetes.io/projected/4070bffc-3574-4ee1-b444-e405af814c82-kube-api-access-h5jpd\") pod \"4070bffc-3574-4ee1-b444-e405af814c82\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " Oct 30 07:18:44.412129 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.411602 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-workload-certs\") pod \"012bfdea-4eb7-442d-a951-d1fd23882318\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " Oct 30 07:18:44.412129 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.411634 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-credential-socket\") pod \"012bfdea-4eb7-442d-a951-d1fd23882318\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " Oct 30 07:18:44.412129 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.411662 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4070bffc-3574-4ee1-b444-e405af814c82-istio-token\" (UniqueName: \"kubernetes.io/projected/4070bffc-3574-4ee1-b444-e405af814c82-istio-token\") pod \"4070bffc-3574-4ee1-b444-e405af814c82\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " Oct 30 07:18:44.412129 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.411722 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-istio-data\") pod \"7e1deed7-8d57-42b0-8277-96ad692ca910\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " Oct 30 07:18:44.412129 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.411754 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-workload-socket\") pod \"4070bffc-3574-4ee1-b444-e405af814c82\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " Oct 30 07:18:44.412129 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.411795 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4070bffc-3574-4ee1-b444-e405af814c82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4070bffc-3574-4ee1-b444-e405af814c82-istiod-ca-cert\") pod \"4070bffc-3574-4ee1-b444-e405af814c82\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " Oct 30 07:18:44.412129 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.411866 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-workload-socket\") pod \"012bfdea-4eb7-442d-a951-d1fd23882318\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " Oct 30 07:18:44.413124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.412644 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:18:44.416304 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.414643 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-workload-certs\") pod \"7e1deed7-8d57-42b0-8277-96ad692ca910\" (UID: \"7e1deed7-8d57-42b0-8277-96ad692ca910\") " Oct 30 07:18:44.416304 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.414706 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/012bfdea-4eb7-442d-a951-d1fd23882318-istio-token\" (UniqueName: \"kubernetes.io/projected/012bfdea-4eb7-442d-a951-d1fd23882318-istio-token\") pod \"012bfdea-4eb7-442d-a951-d1fd23882318\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " Oct 30 07:18:44.416304 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.414733 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-credential-socket\") pod \"4070bffc-3574-4ee1-b444-e405af814c82\" (UID: \"4070bffc-3574-4ee1-b444-e405af814c82\") " Oct 30 07:18:44.416304 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.414821 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-istio-data\") pod \"012bfdea-4eb7-442d-a951-d1fd23882318\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " Oct 30 07:18:44.416304 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.414863 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-istio-envoy\") pod \"012bfdea-4eb7-442d-a951-d1fd23882318\" (UID: \"012bfdea-4eb7-442d-a951-d1fd23882318\") " Oct 30 07:18:44.416304 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.415903 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.416304 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.415928 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdwzx\" (UniqueName: \"kubernetes.io/projected/67817544-1eb4-4e44-ba6c-b84c63df5f76-kube-api-access-mdwzx\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.416304 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.415945 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e1deed7-8d57-42b0-8277-96ad692ca910-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.416304 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.415960 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.416304 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.415977 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.416304 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.415994 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.416304 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.416008 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67817544-1eb4-4e44-ba6c-b84c63df5f76-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.416304 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.416022 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67817544-1eb4-4e44-ba6c-b84c63df5f76-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.416304 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.416036 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67817544-1eb4-4e44-ba6c-b84c63df5f76-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.416304 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.416052 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67817544-1eb4-4e44-ba6c-b84c63df5f76-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.417148 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.411659 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-workload-certs" pod "4070bffc-3574-4ee1-b444-e405af814c82" (UID: "4070bffc-3574-4ee1-b444-e405af814c82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.417148 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.413012 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-workload-socket" pod "7e1deed7-8d57-42b0-8277-96ad692ca910" (UID: "7e1deed7-8d57-42b0-8277-96ad692ca910"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.417148 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.413230 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-workload-socket" pod "012bfdea-4eb7-442d-a951-d1fd23882318" (UID: "012bfdea-4eb7-442d-a951-d1fd23882318"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.417148 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.413428 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-workload-certs" pod "012bfdea-4eb7-442d-a951-d1fd23882318" (UID: "012bfdea-4eb7-442d-a951-d1fd23882318"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.417148 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.413625 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-credential-socket" pod "012bfdea-4eb7-442d-a951-d1fd23882318" (UID: "012bfdea-4eb7-442d-a951-d1fd23882318"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.417148 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.414296 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4070bffc-3574-4ee1-b444-e405af814c82-kube-api-access-h5jpd" pod "4070bffc-3574-4ee1-b444-e405af814c82" (UID: "4070bffc-3574-4ee1-b444-e405af814c82"). InnerVolumeSpecName "kube-api-access-h5jpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:44.417148 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.414382 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-credential-socket" pod "7e1deed7-8d57-42b0-8277-96ad692ca910" (UID: "7e1deed7-8d57-42b0-8277-96ad692ca910"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.417148 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.414613 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/012bfdea-4eb7-442d-a951-d1fd23882318-istiod-ca-cert" pod "012bfdea-4eb7-442d-a951-d1fd23882318" (UID: "012bfdea-4eb7-442d-a951-d1fd23882318"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:18:44.417148 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.416448 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012bfdea-4eb7-442d-a951-d1fd23882318-kube-api-access-f2wcm" pod "012bfdea-4eb7-442d-a951-d1fd23882318" (UID: "012bfdea-4eb7-442d-a951-d1fd23882318"). InnerVolumeSpecName "kube-api-access-f2wcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:44.417148 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.416733 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-istio-data" pod "7e1deed7-8d57-42b0-8277-96ad692ca910" (UID: "7e1deed7-8d57-42b0-8277-96ad692ca910"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.417148 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.417005 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-istio-data" pod "4070bffc-3574-4ee1-b444-e405af814c82" (UID: "4070bffc-3574-4ee1-b444-e405af814c82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.418672 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.417905 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e1deed7-8d57-42b0-8277-96ad692ca910-istio-podinfo" pod "7e1deed7-8d57-42b0-8277-96ad692ca910" (UID: "7e1deed7-8d57-42b0-8277-96ad692ca910"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:18:44.418672 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.417976 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-istio-envoy" pod "4070bffc-3574-4ee1-b444-e405af814c82" (UID: "4070bffc-3574-4ee1-b444-e405af814c82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.418672 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.418146 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-workload-certs" pod "7e1deed7-8d57-42b0-8277-96ad692ca910" (UID: "7e1deed7-8d57-42b0-8277-96ad692ca910"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.418672 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.418352 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-workload-socket" pod "4070bffc-3574-4ee1-b444-e405af814c82" (UID: "4070bffc-3574-4ee1-b444-e405af814c82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.418672 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.418546 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-credential-socket" pod "4070bffc-3574-4ee1-b444-e405af814c82" (UID: "4070bffc-3574-4ee1-b444-e405af814c82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.418672 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.418618 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-istio-envoy" pod "7e1deed7-8d57-42b0-8277-96ad692ca910" (UID: "7e1deed7-8d57-42b0-8277-96ad692ca910"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.419192 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.418933 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4070bffc-3574-4ee1-b444-e405af814c82-istiod-ca-cert" pod "4070bffc-3574-4ee1-b444-e405af814c82" (UID: "4070bffc-3574-4ee1-b444-e405af814c82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:18:44.419192 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.419168 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-istio-data" pod "012bfdea-4eb7-442d-a951-d1fd23882318" (UID: "012bfdea-4eb7-442d-a951-d1fd23882318"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.419655 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.419622 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e1deed7-8d57-42b0-8277-96ad692ca910-istio-token" pod "7e1deed7-8d57-42b0-8277-96ad692ca910" (UID: "7e1deed7-8d57-42b0-8277-96ad692ca910"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:44.419799 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.419757 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4070bffc-3574-4ee1-b444-e405af814c82-istio-token" pod "4070bffc-3574-4ee1-b444-e405af814c82" (UID: "4070bffc-3574-4ee1-b444-e405af814c82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:44.420078 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.420047 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4070bffc-3574-4ee1-b444-e405af814c82-istio-podinfo" pod "4070bffc-3574-4ee1-b444-e405af814c82" (UID: "4070bffc-3574-4ee1-b444-e405af814c82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:18:44.421249 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.421081 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012bfdea-4eb7-442d-a951-d1fd23882318-istio-token" pod "012bfdea-4eb7-442d-a951-d1fd23882318" (UID: "012bfdea-4eb7-442d-a951-d1fd23882318"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:44.421249 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.421168 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/012bfdea-4eb7-442d-a951-d1fd23882318-istio-podinfo" pod "012bfdea-4eb7-442d-a951-d1fd23882318" (UID: "012bfdea-4eb7-442d-a951-d1fd23882318"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:18:44.421249 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.421198 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e1deed7-8d57-42b0-8277-96ad692ca910-kube-api-access-6pwnt" pod "7e1deed7-8d57-42b0-8277-96ad692ca910" (UID: "7e1deed7-8d57-42b0-8277-96ad692ca910"). InnerVolumeSpecName "kube-api-access-6pwnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:44.421249 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.421233 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-istio-envoy" pod "012bfdea-4eb7-442d-a951-d1fd23882318" (UID: "012bfdea-4eb7-442d-a951-d1fd23882318"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.516888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.516858 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-workload-certs\") pod \"745992af-9150-4336-bf13-2e0f087214a5\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " Oct 30 07:18:44.517005 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.516938 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-workload-certs\") pod \"c193cba6-1789-4935-835b-80afc22f58ad\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " Oct 30 07:18:44.517005 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.516986 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-istio-envoy\") pod \"c193cba6-1789-4935-835b-80afc22f58ad\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " Oct 30 07:18:44.517086 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.517015 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-workload-socket\") pod \"c193cba6-1789-4935-835b-80afc22f58ad\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " Oct 30 07:18:44.517086 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.517044 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c193cba6-1789-4935-835b-80afc22f58ad-kube-api-access-55jsx\" (UniqueName: \"kubernetes.io/projected/c193cba6-1789-4935-835b-80afc22f58ad-kube-api-access-55jsx\") pod \"c193cba6-1789-4935-835b-80afc22f58ad\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " Oct 30 07:18:44.517086 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.517061 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/745992af-9150-4336-bf13-2e0f087214a5-kube-api-access-xxdcg\" (UniqueName: \"kubernetes.io/projected/745992af-9150-4336-bf13-2e0f087214a5-kube-api-access-xxdcg\") pod \"745992af-9150-4336-bf13-2e0f087214a5\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " Oct 30 07:18:44.517086 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.517082 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c193cba6-1789-4935-835b-80afc22f58ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c193cba6-1789-4935-835b-80afc22f58ad-istiod-ca-cert\") pod \"c193cba6-1789-4935-835b-80afc22f58ad\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " Oct 30 07:18:44.517215 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.517121 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-workload-certs" pod "745992af-9150-4336-bf13-2e0f087214a5" (UID: "745992af-9150-4336-bf13-2e0f087214a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.517215 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.517135 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/745992af-9150-4336-bf13-2e0f087214a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/745992af-9150-4336-bf13-2e0f087214a5-istio-podinfo\") pod \"745992af-9150-4336-bf13-2e0f087214a5\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " Oct 30 07:18:44.517215 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.517170 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c193cba6-1789-4935-835b-80afc22f58ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c193cba6-1789-4935-835b-80afc22f58ad-istio-podinfo\") pod \"c193cba6-1789-4935-835b-80afc22f58ad\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " Oct 30 07:18:44.517308 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.517221 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/745992af-9150-4336-bf13-2e0f087214a5-istio-token\" (UniqueName: \"kubernetes.io/projected/745992af-9150-4336-bf13-2e0f087214a5-istio-token\") pod \"745992af-9150-4336-bf13-2e0f087214a5\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " Oct 30 07:18:44.517308 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.517270 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/745992af-9150-4336-bf13-2e0f087214a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/745992af-9150-4336-bf13-2e0f087214a5-istiod-ca-cert\") pod \"745992af-9150-4336-bf13-2e0f087214a5\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " Oct 30 07:18:44.517308 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.517294 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-istio-envoy\") pod \"745992af-9150-4336-bf13-2e0f087214a5\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " Oct 30 07:18:44.517397 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.517312 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-istio-data\") pod \"745992af-9150-4336-bf13-2e0f087214a5\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " Oct 30 07:18:44.517397 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.517339 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-credential-socket\") pod \"745992af-9150-4336-bf13-2e0f087214a5\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " Oct 30 07:18:44.517397 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.517362 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-workload-socket\") pod \"745992af-9150-4336-bf13-2e0f087214a5\" (UID: \"745992af-9150-4336-bf13-2e0f087214a5\") " Oct 30 07:18:44.518616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.517496 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c193cba6-1789-4935-835b-80afc22f58ad-istio-token\" (UniqueName: \"kubernetes.io/projected/c193cba6-1789-4935-835b-80afc22f58ad-istio-token\") pod \"c193cba6-1789-4935-835b-80afc22f58ad\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " Oct 30 07:18:44.518616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.517529 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-credential-socket\") pod \"c193cba6-1789-4935-835b-80afc22f58ad\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " Oct 30 07:18:44.518616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.517558 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-istio-data\") pod \"c193cba6-1789-4935-835b-80afc22f58ad\" (UID: \"c193cba6-1789-4935-835b-80afc22f58ad\") " Oct 30 07:18:44.518616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.518075 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.518616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.518095 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012bfdea-4eb7-442d-a951-d1fd23882318-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.518616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.518111 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5jpd\" (UniqueName: \"kubernetes.io/projected/4070bffc-3574-4ee1-b444-e405af814c82-kube-api-access-h5jpd\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.518616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.518133 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.518616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.518147 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.518616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.518170 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4070bffc-3574-4ee1-b444-e405af814c82-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.518616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.518186 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.518616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.518199 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.518616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.518213 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4070bffc-3574-4ee1-b444-e405af814c82-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.518616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.518227 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.518616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.518239 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.518616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.518252 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012bfdea-4eb7-442d-a951-d1fd23882318-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.518616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.518265 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.518616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.518278 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.518616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.518291 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012bfdea-4eb7-442d-a951-d1fd23882318-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.518616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.518305 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e1deed7-8d57-42b0-8277-96ad692ca910-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.518616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.518319 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2wcm\" (UniqueName: \"kubernetes.io/projected/012bfdea-4eb7-442d-a951-d1fd23882318-kube-api-access-f2wcm\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.518616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.518334 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.518616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.517271 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-workload-socket" pod "c193cba6-1789-4935-835b-80afc22f58ad" (UID: "c193cba6-1789-4935-835b-80afc22f58ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.518616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.517398 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-workload-certs" pod "c193cba6-1789-4935-835b-80afc22f58ad" (UID: "c193cba6-1789-4935-835b-80afc22f58ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.518616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.518174 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c193cba6-1789-4935-835b-80afc22f58ad-istiod-ca-cert" pod "c193cba6-1789-4935-835b-80afc22f58ad" (UID: "c193cba6-1789-4935-835b-80afc22f58ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:18:44.518616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.518368 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-istio-data" pod "745992af-9150-4336-bf13-2e0f087214a5" (UID: "745992af-9150-4336-bf13-2e0f087214a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.520554 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.518667 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/745992af-9150-4336-bf13-2e0f087214a5-istiod-ca-cert" pod "745992af-9150-4336-bf13-2e0f087214a5" (UID: "745992af-9150-4336-bf13-2e0f087214a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:18:44.520554 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.518933 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4070bffc-3574-4ee1-b444-e405af814c82-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.520554 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.518954 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.520554 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.518970 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e1deed7-8d57-42b0-8277-96ad692ca910-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.520554 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.518983 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.520554 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.518998 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.520554 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.519012 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4070bffc-3574-4ee1-b444-e405af814c82-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.520554 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.519026 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pwnt\" (UniqueName: \"kubernetes.io/projected/7e1deed7-8d57-42b0-8277-96ad692ca910-kube-api-access-6pwnt\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.520554 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.519042 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1deed7-8d57-42b0-8277-96ad692ca910-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.520554 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.519057 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012bfdea-4eb7-442d-a951-d1fd23882318-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.520554 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.519219 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-workload-socket" pod "745992af-9150-4336-bf13-2e0f087214a5" (UID: "745992af-9150-4336-bf13-2e0f087214a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.520554 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.519426 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-credential-socket" pod "c193cba6-1789-4935-835b-80afc22f58ad" (UID: "c193cba6-1789-4935-835b-80afc22f58ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.520554 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.519454 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-credential-socket" pod "745992af-9150-4336-bf13-2e0f087214a5" (UID: "745992af-9150-4336-bf13-2e0f087214a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.520554 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.519471 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-istio-data" pod "c193cba6-1789-4935-835b-80afc22f58ad" (UID: "c193cba6-1789-4935-835b-80afc22f58ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.521484 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.521452 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/745992af-9150-4336-bf13-2e0f087214a5-kube-api-access-xxdcg" pod "745992af-9150-4336-bf13-2e0f087214a5" (UID: "745992af-9150-4336-bf13-2e0f087214a5"). InnerVolumeSpecName "kube-api-access-xxdcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:44.531628 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.530117 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:18:44.539622 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.535758 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c193cba6-1789-4935-835b-80afc22f58ad-istio-token" pod "c193cba6-1789-4935-835b-80afc22f58ad" (UID: "c193cba6-1789-4935-835b-80afc22f58ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:44.543441 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.542207 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-istio-envoy" pod "745992af-9150-4336-bf13-2e0f087214a5" (UID: "745992af-9150-4336-bf13-2e0f087214a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.593296 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.593186 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-istio-envoy" pod "c193cba6-1789-4935-835b-80afc22f58ad" (UID: "c193cba6-1789-4935-835b-80afc22f58ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.593380 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.593355 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/745992af-9150-4336-bf13-2e0f087214a5-istio-token" pod "745992af-9150-4336-bf13-2e0f087214a5" (UID: "745992af-9150-4336-bf13-2e0f087214a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:44.593627 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.593459 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c193cba6-1789-4935-835b-80afc22f58ad-kube-api-access-55jsx" pod "c193cba6-1789-4935-835b-80afc22f58ad" (UID: "c193cba6-1789-4935-835b-80afc22f58ad"). InnerVolumeSpecName "kube-api-access-55jsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:44.593627 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.593524 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/745992af-9150-4336-bf13-2e0f087214a5-istio-podinfo" pod "745992af-9150-4336-bf13-2e0f087214a5" (UID: "745992af-9150-4336-bf13-2e0f087214a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:18:44.593627 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.593614 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c193cba6-1789-4935-835b-80afc22f58ad-istio-podinfo" pod "c193cba6-1789-4935-835b-80afc22f58ad" (UID: "c193cba6-1789-4935-835b-80afc22f58ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:18:44.636244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.622086 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-istio-envoy\") pod \"1007f3bc-edf3-41a8-b5cf-00bac3178407\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " Oct 30 07:18:44.636244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.622185 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-credential-socket\") pod \"1007f3bc-edf3-41a8-b5cf-00bac3178407\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " Oct 30 07:18:44.636244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.622216 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-workload-certs\") pod \"1007f3bc-edf3-41a8-b5cf-00bac3178407\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " Oct 30 07:18:44.636244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.622262 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1007f3bc-edf3-41a8-b5cf-00bac3178407-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1007f3bc-edf3-41a8-b5cf-00bac3178407-istio-podinfo\") pod \"1007f3bc-edf3-41a8-b5cf-00bac3178407\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " Oct 30 07:18:44.636244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.622307 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1007f3bc-edf3-41a8-b5cf-00bac3178407-istio-token\" (UniqueName: \"kubernetes.io/projected/1007f3bc-edf3-41a8-b5cf-00bac3178407-istio-token\") pod \"1007f3bc-edf3-41a8-b5cf-00bac3178407\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " Oct 30 07:18:44.636244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.622497 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-istio-data\") pod \"1007f3bc-edf3-41a8-b5cf-00bac3178407\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " Oct 30 07:18:44.636244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.622547 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1007f3bc-edf3-41a8-b5cf-00bac3178407-kube-api-access-wxjk4\" (UniqueName: \"kubernetes.io/projected/1007f3bc-edf3-41a8-b5cf-00bac3178407-kube-api-access-wxjk4\") pod \"1007f3bc-edf3-41a8-b5cf-00bac3178407\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " Oct 30 07:18:44.636244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.622631 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1007f3bc-edf3-41a8-b5cf-00bac3178407-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1007f3bc-edf3-41a8-b5cf-00bac3178407-istiod-ca-cert\") pod \"1007f3bc-edf3-41a8-b5cf-00bac3178407\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " Oct 30 07:18:44.636244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.622680 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-workload-socket\") pod \"1007f3bc-edf3-41a8-b5cf-00bac3178407\" (UID: \"1007f3bc-edf3-41a8-b5cf-00bac3178407\") " Oct 30 07:18:44.636244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.623740 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.636244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.623763 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.636244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.623788 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.636244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.623802 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55jsx\" (UniqueName: \"kubernetes.io/projected/c193cba6-1789-4935-835b-80afc22f58ad-kube-api-access-55jsx\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.636244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.623823 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxdcg\" (UniqueName: \"kubernetes.io/projected/745992af-9150-4336-bf13-2e0f087214a5-kube-api-access-xxdcg\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.636244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.623838 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c193cba6-1789-4935-835b-80afc22f58ad-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.636244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.623852 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/745992af-9150-4336-bf13-2e0f087214a5-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.636244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.623865 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c193cba6-1789-4935-835b-80afc22f58ad-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.636244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.623884 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/745992af-9150-4336-bf13-2e0f087214a5-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.636244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.623897 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/745992af-9150-4336-bf13-2e0f087214a5-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.636244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.623910 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.636244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.623922 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.636244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.623942 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.636244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.623957 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/745992af-9150-4336-bf13-2e0f087214a5-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.636244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.623971 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c193cba6-1789-4935-835b-80afc22f58ad-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.636244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.623984 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.636244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.624002 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c193cba6-1789-4935-835b-80afc22f58ad-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.636244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.628030 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-workload-certs" pod "1007f3bc-edf3-41a8-b5cf-00bac3178407" (UID: "1007f3bc-edf3-41a8-b5cf-00bac3178407"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.636244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.628217 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-istio-data" pod "1007f3bc-edf3-41a8-b5cf-00bac3178407" (UID: "1007f3bc-edf3-41a8-b5cf-00bac3178407"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.636244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.628553 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-credential-socket" pod "1007f3bc-edf3-41a8-b5cf-00bac3178407" (UID: "1007f3bc-edf3-41a8-b5cf-00bac3178407"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.636244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.629600 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-workload-socket" pod "1007f3bc-edf3-41a8-b5cf-00bac3178407" (UID: "1007f3bc-edf3-41a8-b5cf-00bac3178407"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.681169 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.636482 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1007f3bc-edf3-41a8-b5cf-00bac3178407-istiod-ca-cert" pod "1007f3bc-edf3-41a8-b5cf-00bac3178407" (UID: "1007f3bc-edf3-41a8-b5cf-00bac3178407"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:18:44.681169 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.675187 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1007f3bc-edf3-41a8-b5cf-00bac3178407-istio-token" pod "1007f3bc-edf3-41a8-b5cf-00bac3178407" (UID: "1007f3bc-edf3-41a8-b5cf-00bac3178407"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:44.681169 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.676753 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-istio-envoy" pod "1007f3bc-edf3-41a8-b5cf-00bac3178407" (UID: "1007f3bc-edf3-41a8-b5cf-00bac3178407"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.690722 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.690660 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1007f3bc-edf3-41a8-b5cf-00bac3178407-kube-api-access-wxjk4" pod "1007f3bc-edf3-41a8-b5cf-00bac3178407" (UID: "1007f3bc-edf3-41a8-b5cf-00bac3178407"). InnerVolumeSpecName "kube-api-access-wxjk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:44.691007 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.690958 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1007f3bc-edf3-41a8-b5cf-00bac3178407-istio-podinfo" pod "1007f3bc-edf3-41a8-b5cf-00bac3178407" (UID: "1007f3bc-edf3-41a8-b5cf-00bac3178407"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:18:44.715189 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.715106 4951 generic.go:356] "Generic (PLEG): container finished" podID="7e1deed7-8d57-42b0-8277-96ad692ca910" containerID="961594a7e4af9c81a6ce86df5f263720e9bc9723f7a6f0ed61821d248038ce50" exitCode=0 Oct 30 07:18:44.715701 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.715175 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" event={"ID":"7e1deed7-8d57-42b0-8277-96ad692ca910","Type":"ContainerDied","Data":"961594a7e4af9c81a6ce86df5f263720e9bc9723f7a6f0ed61821d248038ce50"} Oct 30 07:18:44.715701 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.715239 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" Oct 30 07:18:44.715701 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.715417 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns" event={"ID":"7e1deed7-8d57-42b0-8277-96ad692ca910","Type":"ContainerDied","Data":"330b5cac47e3a27001495c2b5bed949a6d04b3689789915765cfff4e37cae9d2"} Oct 30 07:18:44.715701 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.715464 4951 scope.go:122] "RemoveContainer" containerID="961594a7e4af9c81a6ce86df5f263720e9bc9723f7a6f0ed61821d248038ce50" Oct 30 07:18:44.718562 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.718005 4951 generic.go:356] "Generic (PLEG): container finished" podID="1007f3bc-edf3-41a8-b5cf-00bac3178407" containerID="728ed508cb0b801bc2352dfd313ba08a587530d2a9b97b6994d6e333276b4c38" exitCode=0 Oct 30 07:18:44.718562 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.718088 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" Oct 30 07:18:44.718562 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.718082 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" event={"ID":"1007f3bc-edf3-41a8-b5cf-00bac3178407","Type":"ContainerDied","Data":"728ed508cb0b801bc2352dfd313ba08a587530d2a9b97b6994d6e333276b4c38"} Oct 30 07:18:44.718562 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.718196 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td" event={"ID":"1007f3bc-edf3-41a8-b5cf-00bac3178407","Type":"ContainerDied","Data":"16806b000361980dd0454bef65c9fa963789bccfaaf7c037456758335e4ce4f5"} Oct 30 07:18:44.735794 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.719934 4951 generic.go:356] "Generic (PLEG): container finished" podID="c193cba6-1789-4935-835b-80afc22f58ad" containerID="f456dd4a2e31710d2c1418b6b27d835ebb38edcb7d51c8e26c04fbdae0af7f5c" exitCode=0 Oct 30 07:18:44.735794 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.720006 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" event={"ID":"c193cba6-1789-4935-835b-80afc22f58ad","Type":"ContainerDied","Data":"f456dd4a2e31710d2c1418b6b27d835ebb38edcb7d51c8e26c04fbdae0af7f5c"} Oct 30 07:18:44.735794 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.720038 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" event={"ID":"c193cba6-1789-4935-835b-80afc22f58ad","Type":"ContainerDied","Data":"01387206b1ce8a89a81239f5072276167d11ab18f3930e360e7f2ac41737cf42"} Oct 30 07:18:44.735794 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.720111 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j" Oct 30 07:18:44.735794 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.722171 4951 generic.go:356] "Generic (PLEG): container finished" podID="67817544-1eb4-4e44-ba6c-b84c63df5f76" containerID="d13f424ae56f9715ca3ff73c2ed395644125fa23e08d9a093dfaf9bf0971b704" exitCode=0 Oct 30 07:18:44.735794 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.722259 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" event={"ID":"67817544-1eb4-4e44-ba6c-b84c63df5f76","Type":"ContainerDied","Data":"d13f424ae56f9715ca3ff73c2ed395644125fa23e08d9a093dfaf9bf0971b704"} Oct 30 07:18:44.735794 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.722302 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" event={"ID":"67817544-1eb4-4e44-ba6c-b84c63df5f76","Type":"ContainerDied","Data":"43c9af56b50449a557a67c063aad5638c6fb82fdb60d3e3be834a28bc5845c2c"} Oct 30 07:18:44.735794 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.722395 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9" Oct 30 07:18:44.735794 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.725838 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.735794 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.725863 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.735794 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.725877 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.735794 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.725891 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1007f3bc-edf3-41a8-b5cf-00bac3178407-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.735794 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.725903 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1007f3bc-edf3-41a8-b5cf-00bac3178407-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.735794 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.725916 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.735794 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.725929 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxjk4\" (UniqueName: \"kubernetes.io/projected/1007f3bc-edf3-41a8-b5cf-00bac3178407-kube-api-access-wxjk4\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.735794 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.725943 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1007f3bc-edf3-41a8-b5cf-00bac3178407-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.735794 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.725956 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1007f3bc-edf3-41a8-b5cf-00bac3178407-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.777241 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.772863 4951 patch_prober.go:29] interesting pod/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 07:18:44.777241 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.772911 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" podUID="f3d3f735-6cd0-4153-a6ff-88f77f4941ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 07:18:44.777241 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.773173 4951 generic.go:356] "Generic (PLEG): container finished" podID="745992af-9150-4336-bf13-2e0f087214a5" containerID="3343d0803013b4629c2128935e9174ec7feb1a7fefeede37759adb5a3284d980" exitCode=0 Oct 30 07:18:44.777241 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.773283 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" event={"ID":"745992af-9150-4336-bf13-2e0f087214a5","Type":"ContainerDied","Data":"3343d0803013b4629c2128935e9174ec7feb1a7fefeede37759adb5a3284d980"} Oct 30 07:18:44.777241 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.773321 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" event={"ID":"745992af-9150-4336-bf13-2e0f087214a5","Type":"ContainerDied","Data":"491b2835befd8efbb05f149b3907ec1e5e67ca046fb972bdc8e475513f1d798b"} Oct 30 07:18:44.777241 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.773463 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb" Oct 30 07:18:44.780875 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.780823 4951 generic.go:356] "Generic (PLEG): container finished" podID="4070bffc-3574-4ee1-b444-e405af814c82" containerID="ab77304f3a90305cc11fbe0ba43694d858a760ebe62c02b137ec514d418a6529" exitCode=0 Oct 30 07:18:44.781111 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.780975 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" Oct 30 07:18:44.781111 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.781011 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" event={"ID":"4070bffc-3574-4ee1-b444-e405af814c82","Type":"ContainerDied","Data":"ab77304f3a90305cc11fbe0ba43694d858a760ebe62c02b137ec514d418a6529"} Oct 30 07:18:44.781111 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.781067 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m" event={"ID":"4070bffc-3574-4ee1-b444-e405af814c82","Type":"ContainerDied","Data":"c1e6d08deb24261d1fe8ba5781b56e012344f5b73e91e19273e4a5eb885b0607"} Oct 30 07:18:44.787345 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.783152 4951 generic.go:356] "Generic (PLEG): container finished" podID="012bfdea-4eb7-442d-a951-d1fd23882318" containerID="8c0a7b6f94a9b8c382354be1b87553e07f84fe070e21a8c795e58aef60fe5ac0" exitCode=0 Oct 30 07:18:44.787345 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.783303 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" event={"ID":"012bfdea-4eb7-442d-a951-d1fd23882318","Type":"ContainerDied","Data":"8c0a7b6f94a9b8c382354be1b87553e07f84fe070e21a8c795e58aef60fe5ac0"} Oct 30 07:18:44.787345 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.783371 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" event={"ID":"012bfdea-4eb7-442d-a951-d1fd23882318","Type":"ContainerDied","Data":"fcc3aba3d885e0c87bee1764dbce26399a078e506f486ada4638ea0513db0164"} Oct 30 07:18:44.787345 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.783496 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8" Oct 30 07:18:44.790765 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.790699 4951 scope.go:122] "RemoveContainer" containerID="961594a7e4af9c81a6ce86df5f263720e9bc9723f7a6f0ed61821d248038ce50" Oct 30 07:18:44.791071 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:18:44.790955 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"961594a7e4af9c81a6ce86df5f263720e9bc9723f7a6f0ed61821d248038ce50\": container with ID starting with 961594a7e4af9c81a6ce86df5f263720e9bc9723f7a6f0ed61821d248038ce50 not found: ID does not exist" containerID="961594a7e4af9c81a6ce86df5f263720e9bc9723f7a6f0ed61821d248038ce50" Oct 30 07:18:44.791071 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.790989 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"961594a7e4af9c81a6ce86df5f263720e9bc9723f7a6f0ed61821d248038ce50"} err="failed to get container status \"961594a7e4af9c81a6ce86df5f263720e9bc9723f7a6f0ed61821d248038ce50\": rpc error: code = NotFound desc = could not find container \"961594a7e4af9c81a6ce86df5f263720e9bc9723f7a6f0ed61821d248038ce50\": container with ID starting with 961594a7e4af9c81a6ce86df5f263720e9bc9723f7a6f0ed61821d248038ce50 not found: ID does not exist" Oct 30 07:18:44.791071 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.791008 4951 scope.go:122] "RemoveContainer" containerID="728ed508cb0b801bc2352dfd313ba08a587530d2a9b97b6994d6e333276b4c38" Oct 30 07:18:44.803679 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.803659 4951 scope.go:122] "RemoveContainer" containerID="728ed508cb0b801bc2352dfd313ba08a587530d2a9b97b6994d6e333276b4c38" Oct 30 07:18:44.804023 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:18:44.803967 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"728ed508cb0b801bc2352dfd313ba08a587530d2a9b97b6994d6e333276b4c38\": container with ID starting with 728ed508cb0b801bc2352dfd313ba08a587530d2a9b97b6994d6e333276b4c38 not found: ID does not exist" containerID="728ed508cb0b801bc2352dfd313ba08a587530d2a9b97b6994d6e333276b4c38" Oct 30 07:18:44.804067 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.804029 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"728ed508cb0b801bc2352dfd313ba08a587530d2a9b97b6994d6e333276b4c38"} err="failed to get container status \"728ed508cb0b801bc2352dfd313ba08a587530d2a9b97b6994d6e333276b4c38\": rpc error: code = NotFound desc = could not find container \"728ed508cb0b801bc2352dfd313ba08a587530d2a9b97b6994d6e333276b4c38\": container with ID starting with 728ed508cb0b801bc2352dfd313ba08a587530d2a9b97b6994d6e333276b4c38 not found: ID does not exist" Oct 30 07:18:44.804067 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.804048 4951 scope.go:122] "RemoveContainer" containerID="f456dd4a2e31710d2c1418b6b27d835ebb38edcb7d51c8e26c04fbdae0af7f5c" Oct 30 07:18:44.815474 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.815458 4951 scope.go:122] "RemoveContainer" containerID="f456dd4a2e31710d2c1418b6b27d835ebb38edcb7d51c8e26c04fbdae0af7f5c" Oct 30 07:18:44.815776 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:18:44.815754 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f456dd4a2e31710d2c1418b6b27d835ebb38edcb7d51c8e26c04fbdae0af7f5c\": container with ID starting with f456dd4a2e31710d2c1418b6b27d835ebb38edcb7d51c8e26c04fbdae0af7f5c not found: ID does not exist" containerID="f456dd4a2e31710d2c1418b6b27d835ebb38edcb7d51c8e26c04fbdae0af7f5c" Oct 30 07:18:44.815849 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.815784 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f456dd4a2e31710d2c1418b6b27d835ebb38edcb7d51c8e26c04fbdae0af7f5c"} err="failed to get container status \"f456dd4a2e31710d2c1418b6b27d835ebb38edcb7d51c8e26c04fbdae0af7f5c\": rpc error: code = NotFound desc = could not find container \"f456dd4a2e31710d2c1418b6b27d835ebb38edcb7d51c8e26c04fbdae0af7f5c\": container with ID starting with f456dd4a2e31710d2c1418b6b27d835ebb38edcb7d51c8e26c04fbdae0af7f5c not found: ID does not exist" Oct 30 07:18:44.815849 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.815806 4951 scope.go:122] "RemoveContainer" containerID="d13f424ae56f9715ca3ff73c2ed395644125fa23e08d9a093dfaf9bf0971b704" Oct 30 07:18:44.890532 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.890513 4951 scope.go:122] "RemoveContainer" containerID="d13f424ae56f9715ca3ff73c2ed395644125fa23e08d9a093dfaf9bf0971b704" Oct 30 07:18:44.890960 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:18:44.890940 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d13f424ae56f9715ca3ff73c2ed395644125fa23e08d9a093dfaf9bf0971b704\": container with ID starting with d13f424ae56f9715ca3ff73c2ed395644125fa23e08d9a093dfaf9bf0971b704 not found: ID does not exist" containerID="d13f424ae56f9715ca3ff73c2ed395644125fa23e08d9a093dfaf9bf0971b704" Oct 30 07:18:44.890998 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.890969 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d13f424ae56f9715ca3ff73c2ed395644125fa23e08d9a093dfaf9bf0971b704"} err="failed to get container status \"d13f424ae56f9715ca3ff73c2ed395644125fa23e08d9a093dfaf9bf0971b704\": rpc error: code = NotFound desc = could not find container \"d13f424ae56f9715ca3ff73c2ed395644125fa23e08d9a093dfaf9bf0971b704\": container with ID starting with d13f424ae56f9715ca3ff73c2ed395644125fa23e08d9a093dfaf9bf0971b704 not found: ID does not exist" Oct 30 07:18:44.890998 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.890987 4951 scope.go:122] "RemoveContainer" containerID="3343d0803013b4629c2128935e9174ec7feb1a7fefeede37759adb5a3284d980" Oct 30 07:18:44.901791 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.901709 4951 scope.go:122] "RemoveContainer" containerID="3343d0803013b4629c2128935e9174ec7feb1a7fefeede37759adb5a3284d980" Oct 30 07:18:44.902051 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:18:44.902027 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3343d0803013b4629c2128935e9174ec7feb1a7fefeede37759adb5a3284d980\": container with ID starting with 3343d0803013b4629c2128935e9174ec7feb1a7fefeede37759adb5a3284d980 not found: ID does not exist" containerID="3343d0803013b4629c2128935e9174ec7feb1a7fefeede37759adb5a3284d980" Oct 30 07:18:44.902109 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.902058 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3343d0803013b4629c2128935e9174ec7feb1a7fefeede37759adb5a3284d980"} err="failed to get container status \"3343d0803013b4629c2128935e9174ec7feb1a7fefeede37759adb5a3284d980\": rpc error: code = NotFound desc = could not find container \"3343d0803013b4629c2128935e9174ec7feb1a7fefeede37759adb5a3284d980\": container with ID starting with 3343d0803013b4629c2128935e9174ec7feb1a7fefeede37759adb5a3284d980 not found: ID does not exist" Oct 30 07:18:44.902109 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.902075 4951 scope.go:122] "RemoveContainer" containerID="ab77304f3a90305cc11fbe0ba43694d858a760ebe62c02b137ec514d418a6529" Oct 30 07:18:44.912221 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.912200 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9"] Oct 30 07:18:44.915221 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.915205 4951 scope.go:122] "RemoveContainer" containerID="ab77304f3a90305cc11fbe0ba43694d858a760ebe62c02b137ec514d418a6529" Oct 30 07:18:44.915480 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:18:44.915462 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab77304f3a90305cc11fbe0ba43694d858a760ebe62c02b137ec514d418a6529\": container with ID starting with ab77304f3a90305cc11fbe0ba43694d858a760ebe62c02b137ec514d418a6529 not found: ID does not exist" containerID="ab77304f3a90305cc11fbe0ba43694d858a760ebe62c02b137ec514d418a6529" Oct 30 07:18:44.915555 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.915489 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab77304f3a90305cc11fbe0ba43694d858a760ebe62c02b137ec514d418a6529"} err="failed to get container status \"ab77304f3a90305cc11fbe0ba43694d858a760ebe62c02b137ec514d418a6529\": rpc error: code = NotFound desc = could not find container \"ab77304f3a90305cc11fbe0ba43694d858a760ebe62c02b137ec514d418a6529\": container with ID starting with ab77304f3a90305cc11fbe0ba43694d858a760ebe62c02b137ec514d418a6529 not found: ID does not exist" Oct 30 07:18:44.915555 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.915511 4951 scope.go:122] "RemoveContainer" containerID="8c0a7b6f94a9b8c382354be1b87553e07f84fe070e21a8c795e58aef60fe5ac0" Oct 30 07:18:44.917277 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.917256 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-m7sx9"] Oct 30 07:18:44.934356 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.934334 4951 scope.go:122] "RemoveContainer" containerID="8c0a7b6f94a9b8c382354be1b87553e07f84fe070e21a8c795e58aef60fe5ac0" Oct 30 07:18:44.936061 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:18:44.936020 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c0a7b6f94a9b8c382354be1b87553e07f84fe070e21a8c795e58aef60fe5ac0\": container with ID starting with 8c0a7b6f94a9b8c382354be1b87553e07f84fe070e21a8c795e58aef60fe5ac0 not found: ID does not exist" containerID="8c0a7b6f94a9b8c382354be1b87553e07f84fe070e21a8c795e58aef60fe5ac0" Oct 30 07:18:44.936254 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.936225 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c0a7b6f94a9b8c382354be1b87553e07f84fe070e21a8c795e58aef60fe5ac0"} err="failed to get container status \"8c0a7b6f94a9b8c382354be1b87553e07f84fe070e21a8c795e58aef60fe5ac0\": rpc error: code = NotFound desc = could not find container \"8c0a7b6f94a9b8c382354be1b87553e07f84fe070e21a8c795e58aef60fe5ac0\": container with ID starting with 8c0a7b6f94a9b8c382354be1b87553e07f84fe070e21a8c795e58aef60fe5ac0 not found: ID does not exist" Oct 30 07:18:44.971736 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.971704 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m"] Oct 30 07:18:44.985253 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.985213 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bxg7m"] Oct 30 07:18:44.994709 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:44.994675 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td"] Oct 30 07:18:45.001198 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:45.001169 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-4q2td"] Oct 30 07:18:45.058685 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:45.058644 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb"] Oct 30 07:18:45.060917 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:45.060890 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-jx6vb"] Oct 30 07:18:45.085384 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:45.085351 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns"] Oct 30 07:18:45.094710 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:45.094676 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-l6tns"] Oct 30 07:18:45.167015 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:45.166926 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8"] Oct 30 07:18:45.172954 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:45.172917 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-2j4w8"] Oct 30 07:18:45.188197 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:45.188171 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j"] Oct 30 07:18:45.192646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:45.192621 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-8p69j"] Oct 30 07:18:45.740372 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:45.730014 4951 patch_prober.go:29] interesting pod/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 07:18:45.740372 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:45.730084 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" podUID="f3d3f735-6cd0-4153-a6ff-88f77f4941ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 07:18:45.776927 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:45.776894 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="012bfdea-4eb7-442d-a951-d1fd23882318" path="/var/lib/kubelet/pods/012bfdea-4eb7-442d-a951-d1fd23882318/volumes" Oct 30 07:18:45.777841 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:45.777816 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1007f3bc-edf3-41a8-b5cf-00bac3178407" path="/var/lib/kubelet/pods/1007f3bc-edf3-41a8-b5cf-00bac3178407/volumes" Oct 30 07:18:45.778638 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:45.778617 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4070bffc-3574-4ee1-b444-e405af814c82" path="/var/lib/kubelet/pods/4070bffc-3574-4ee1-b444-e405af814c82/volumes" Oct 30 07:18:45.779862 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:45.779840 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67817544-1eb4-4e44-ba6c-b84c63df5f76" path="/var/lib/kubelet/pods/67817544-1eb4-4e44-ba6c-b84c63df5f76/volumes" Oct 30 07:18:45.780699 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:45.780681 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="745992af-9150-4336-bf13-2e0f087214a5" path="/var/lib/kubelet/pods/745992af-9150-4336-bf13-2e0f087214a5/volumes" Oct 30 07:18:45.781455 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:45.781434 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e1deed7-8d57-42b0-8277-96ad692ca910" path="/var/lib/kubelet/pods/7e1deed7-8d57-42b0-8277-96ad692ca910/volumes" Oct 30 07:18:45.782668 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:45.782647 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c193cba6-1789-4935-835b-80afc22f58ad" path="/var/lib/kubelet/pods/c193cba6-1789-4935-835b-80afc22f58ad/volumes" Oct 30 07:18:46.729077 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:46.729032 4951 patch_prober.go:29] interesting pod/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 07:18:46.729380 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:46.729344 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" podUID="f3d3f735-6cd0-4153-a6ff-88f77f4941ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 07:18:47.730923 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:47.730886 4951 patch_prober.go:29] interesting pod/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 07:18:47.731513 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:47.731475 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" podUID="f3d3f735-6cd0-4153-a6ff-88f77f4941ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 07:18:48.422256 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:48.422222 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-318524-65648b9ff6-7dz6g"] Oct 30 07:18:48.433602 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:48.433369 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-318524-65648b9ff6-7dz6g" podUID="cdad6c53-2e5e-4f0f-b5a5-598b27715e11" containerName="maas-api" containerID="cri-o://a2ea7bb317399579db4a11e37d2b854f93e40f9b8411ca65cf43e7dedfcc3556" gracePeriod=30 Oct 30 07:18:48.590610 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:18:48.590566 4951 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdad6c53_2e5e_4f0f_b5a5_598b27715e11.slice/crio-conmon-a2ea7bb317399579db4a11e37d2b854f93e40f9b8411ca65cf43e7dedfcc3556.scope\": RecentStats: unable to find data in memory cache]" Oct 30 07:18:48.736604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:48.733769 4951 patch_prober.go:29] interesting pod/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 07:18:48.736604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:48.733834 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" podUID="f3d3f735-6cd0-4153-a6ff-88f77f4941ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 07:18:49.340196 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:49.340146 4951 generic.go:356] "Generic (PLEG): container finished" podID="cdad6c53-2e5e-4f0f-b5a5-598b27715e11" containerID="a2ea7bb317399579db4a11e37d2b854f93e40f9b8411ca65cf43e7dedfcc3556" exitCode=0 Oct 30 07:18:49.340419 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:49.340307 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-318524-65648b9ff6-7dz6g" event={"ID":"cdad6c53-2e5e-4f0f-b5a5-598b27715e11","Type":"ContainerDied","Data":"a2ea7bb317399579db4a11e37d2b854f93e40f9b8411ca65cf43e7dedfcc3556"} Oct 30 07:18:49.448374 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:49.448338 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-318524-65648b9ff6-7dz6g" Oct 30 07:18:49.576882 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:49.576843 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdad6c53-2e5e-4f0f-b5a5-598b27715e11-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdad6c53-2e5e-4f0f-b5a5-598b27715e11-openshift-service-ca\") pod \"cdad6c53-2e5e-4f0f-b5a5-598b27715e11\" (UID: \"cdad6c53-2e5e-4f0f-b5a5-598b27715e11\") " Oct 30 07:18:49.577091 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:49.576953 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdad6c53-2e5e-4f0f-b5a5-598b27715e11-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdad6c53-2e5e-4f0f-b5a5-598b27715e11-metrics-certs\") pod \"cdad6c53-2e5e-4f0f-b5a5-598b27715e11\" (UID: \"cdad6c53-2e5e-4f0f-b5a5-598b27715e11\") " Oct 30 07:18:49.577091 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:49.576990 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdad6c53-2e5e-4f0f-b5a5-598b27715e11-kube-api-access-ht969\" (UniqueName: \"kubernetes.io/projected/cdad6c53-2e5e-4f0f-b5a5-598b27715e11-kube-api-access-ht969\") pod \"cdad6c53-2e5e-4f0f-b5a5-598b27715e11\" (UID: \"cdad6c53-2e5e-4f0f-b5a5-598b27715e11\") " Oct 30 07:18:49.577091 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:49.577035 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdad6c53-2e5e-4f0f-b5a5-598b27715e11-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdad6c53-2e5e-4f0f-b5a5-598b27715e11-maas-api-tls\") pod \"cdad6c53-2e5e-4f0f-b5a5-598b27715e11\" (UID: \"cdad6c53-2e5e-4f0f-b5a5-598b27715e11\") " Oct 30 07:18:49.577429 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:49.577296 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdad6c53-2e5e-4f0f-b5a5-598b27715e11-openshift-service-ca" pod "cdad6c53-2e5e-4f0f-b5a5-598b27715e11" (UID: "cdad6c53-2e5e-4f0f-b5a5-598b27715e11"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:18:49.577869 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:49.577839 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdad6c53-2e5e-4f0f-b5a5-598b27715e11-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:49.579815 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:49.579787 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdad6c53-2e5e-4f0f-b5a5-598b27715e11-metrics-certs" pod "cdad6c53-2e5e-4f0f-b5a5-598b27715e11" (UID: "cdad6c53-2e5e-4f0f-b5a5-598b27715e11"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:18:49.579962 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:49.579938 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdad6c53-2e5e-4f0f-b5a5-598b27715e11-maas-api-tls" pod "cdad6c53-2e5e-4f0f-b5a5-598b27715e11" (UID: "cdad6c53-2e5e-4f0f-b5a5-598b27715e11"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:18:49.581807 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:49.581779 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdad6c53-2e5e-4f0f-b5a5-598b27715e11-kube-api-access-ht969" pod "cdad6c53-2e5e-4f0f-b5a5-598b27715e11" (UID: "cdad6c53-2e5e-4f0f-b5a5-598b27715e11"). InnerVolumeSpecName "kube-api-access-ht969". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:49.679814 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:49.679776 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdad6c53-2e5e-4f0f-b5a5-598b27715e11-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:49.679814 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:49.679816 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdad6c53-2e5e-4f0f-b5a5-598b27715e11-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:49.680035 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:49.679833 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht969\" (UniqueName: \"kubernetes.io/projected/cdad6c53-2e5e-4f0f-b5a5-598b27715e11-kube-api-access-ht969\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:18:49.733502 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:49.733462 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:18:49.733948 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:49.733925 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:18:49.734551 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:49.734527 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:18:50.350331 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:50.350299 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-318524-65648b9ff6-7dz6g" Oct 30 07:18:50.350862 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:50.350329 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-318524-65648b9ff6-7dz6g" event={"ID":"cdad6c53-2e5e-4f0f-b5a5-598b27715e11","Type":"ContainerDied","Data":"464a83dff8c5527b51c28895f12bc7b4d6649febc324112748335e5dc07f4590"} Oct 30 07:18:50.350862 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:50.350394 4951 scope.go:122] "RemoveContainer" containerID="a2ea7bb317399579db4a11e37d2b854f93e40f9b8411ca65cf43e7dedfcc3556" Oct 30 07:18:50.403037 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:50.403009 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-318524-65648b9ff6-7dz6g"] Oct 30 07:18:50.413664 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:50.413635 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-318524-65648b9ff6-7dz6g"] Oct 30 07:18:50.523370 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:50.523341 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-052254-589746f98c-l74cl" Oct 30 07:18:51.775667 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:51.775637 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdad6c53-2e5e-4f0f-b5a5-598b27715e11" path="/var/lib/kubelet/pods/cdad6c53-2e5e-4f0f-b5a5-598b27715e11/volumes" Oct 30 07:18:53.026186 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:18:53.026143 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:18:54.096420 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.096389 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-052254-bc7487fc6-qw9h7"] Oct 30 07:18:54.097103 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.097086 4951 cpu_manager.go:414] "containerMap: removing container" podUID="012bfdea-4eb7-442d-a951-d1fd23882318" containerName="istio-proxy" Oct 30 07:18:54.097103 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.097104 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="012bfdea-4eb7-442d-a951-d1fd23882318" containerName="istio-proxy" Oct 30 07:18:54.097217 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.097137 4951 cpu_manager.go:414] "containerMap: removing container" podUID="cdad6c53-2e5e-4f0f-b5a5-598b27715e11" containerName="maas-api" Oct 30 07:18:54.097217 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.097143 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdad6c53-2e5e-4f0f-b5a5-598b27715e11" containerName="maas-api" Oct 30 07:18:54.097217 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.097164 4951 cpu_manager.go:414] "containerMap: removing container" podUID="c193cba6-1789-4935-835b-80afc22f58ad" containerName="istio-proxy" Oct 30 07:18:54.097217 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.097169 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c193cba6-1789-4935-835b-80afc22f58ad" containerName="istio-proxy" Oct 30 07:18:54.097217 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.097178 4951 cpu_manager.go:414] "containerMap: removing container" podUID="745992af-9150-4336-bf13-2e0f087214a5" containerName="istio-proxy" Oct 30 07:18:54.097217 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.097182 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="745992af-9150-4336-bf13-2e0f087214a5" containerName="istio-proxy" Oct 30 07:18:54.097217 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.097196 4951 cpu_manager.go:414] "containerMap: removing container" podUID="4070bffc-3574-4ee1-b444-e405af814c82" containerName="istio-proxy" Oct 30 07:18:54.097217 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.097200 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4070bffc-3574-4ee1-b444-e405af814c82" containerName="istio-proxy" Oct 30 07:18:54.097217 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.097216 4951 cpu_manager.go:414] "containerMap: removing container" podUID="7e1deed7-8d57-42b0-8277-96ad692ca910" containerName="istio-proxy" Oct 30 07:18:54.097217 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.097222 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e1deed7-8d57-42b0-8277-96ad692ca910" containerName="istio-proxy" Oct 30 07:18:54.097504 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.097232 4951 cpu_manager.go:414] "containerMap: removing container" podUID="67817544-1eb4-4e44-ba6c-b84c63df5f76" containerName="istio-proxy" Oct 30 07:18:54.097504 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.097236 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67817544-1eb4-4e44-ba6c-b84c63df5f76" containerName="istio-proxy" Oct 30 07:18:54.097504 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.097244 4951 cpu_manager.go:414] "containerMap: removing container" podUID="1007f3bc-edf3-41a8-b5cf-00bac3178407" containerName="istio-proxy" Oct 30 07:18:54.097504 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.097249 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1007f3bc-edf3-41a8-b5cf-00bac3178407" containerName="istio-proxy" Oct 30 07:18:54.097504 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.097340 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="c193cba6-1789-4935-835b-80afc22f58ad" containerName="istio-proxy" Oct 30 07:18:54.097504 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.097350 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="745992af-9150-4336-bf13-2e0f087214a5" containerName="istio-proxy" Oct 30 07:18:54.097504 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.097357 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="4070bffc-3574-4ee1-b444-e405af814c82" containerName="istio-proxy" Oct 30 07:18:54.097504 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.097367 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e1deed7-8d57-42b0-8277-96ad692ca910" containerName="istio-proxy" Oct 30 07:18:54.097504 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.097374 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="67817544-1eb4-4e44-ba6c-b84c63df5f76" containerName="istio-proxy" Oct 30 07:18:54.097504 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.097380 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="1007f3bc-edf3-41a8-b5cf-00bac3178407" containerName="istio-proxy" Oct 30 07:18:54.097504 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.097386 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="012bfdea-4eb7-442d-a951-d1fd23882318" containerName="istio-proxy" Oct 30 07:18:54.097504 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.097394 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdad6c53-2e5e-4f0f-b5a5-598b27715e11" containerName="maas-api" Oct 30 07:18:54.101153 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.101137 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-052254-bc7487fc6-qw9h7" Oct 30 07:18:54.145321 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.145277 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-052254-bc7487fc6-qw9h7"] Oct 30 07:18:54.145525 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.145468 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-052254-serving-cert\"" Oct 30 07:18:54.164274 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.164232 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fdb2170-1155-429b-87e9-6827bdafbdee-metrics-certs\") pod \"maas-api-e2e-api-b-052254-bc7487fc6-qw9h7\" (UID: \"0fdb2170-1155-429b-87e9-6827bdafbdee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-052254-bc7487fc6-qw9h7" Oct 30 07:18:54.164459 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.164365 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fdb2170-1155-429b-87e9-6827bdafbdee-maas-api-tls\") pod \"maas-api-e2e-api-b-052254-bc7487fc6-qw9h7\" (UID: \"0fdb2170-1155-429b-87e9-6827bdafbdee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-052254-bc7487fc6-qw9h7" Oct 30 07:18:54.164559 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.164469 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fdb2170-1155-429b-87e9-6827bdafbdee-openshift-service-ca\") pod \"maas-api-e2e-api-b-052254-bc7487fc6-qw9h7\" (UID: \"0fdb2170-1155-429b-87e9-6827bdafbdee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-052254-bc7487fc6-qw9h7" Oct 30 07:18:54.164672 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.164611 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm654\" (UniqueName: \"kubernetes.io/projected/0fdb2170-1155-429b-87e9-6827bdafbdee-kube-api-access-jm654\") pod \"maas-api-e2e-api-b-052254-bc7487fc6-qw9h7\" (UID: \"0fdb2170-1155-429b-87e9-6827bdafbdee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-052254-bc7487fc6-qw9h7" Oct 30 07:18:54.224191 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.223819 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5"] Oct 30 07:18:54.230094 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.230063 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:18:54.253560 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.253522 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-0cd8f0-openshift-default-dockercfg-n9d98\"" Oct 30 07:18:54.263004 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.262191 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5"] Oct 30 07:18:54.268148 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.266226 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fdb2170-1155-429b-87e9-6827bdafbdee-metrics-certs\") pod \"maas-api-e2e-api-b-052254-bc7487fc6-qw9h7\" (UID: \"0fdb2170-1155-429b-87e9-6827bdafbdee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-052254-bc7487fc6-qw9h7" Oct 30 07:18:54.268148 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.266347 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fdb2170-1155-429b-87e9-6827bdafbdee-maas-api-tls\") pod \"maas-api-e2e-api-b-052254-bc7487fc6-qw9h7\" (UID: \"0fdb2170-1155-429b-87e9-6827bdafbdee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-052254-bc7487fc6-qw9h7" Oct 30 07:18:54.268148 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.266450 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fdb2170-1155-429b-87e9-6827bdafbdee-openshift-service-ca\") pod \"maas-api-e2e-api-b-052254-bc7487fc6-qw9h7\" (UID: \"0fdb2170-1155-429b-87e9-6827bdafbdee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-052254-bc7487fc6-qw9h7" Oct 30 07:18:54.268148 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.266593 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm654\" (UniqueName: \"kubernetes.io/projected/0fdb2170-1155-429b-87e9-6827bdafbdee-kube-api-access-jm654\") pod \"maas-api-e2e-api-b-052254-bc7487fc6-qw9h7\" (UID: \"0fdb2170-1155-429b-87e9-6827bdafbdee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-052254-bc7487fc6-qw9h7" Oct 30 07:18:54.282142 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:18:54.277666 4951 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-052254-serving-cert: secret "maas-api-e2e-api-b-052254-serving-cert" not found Oct 30 07:18:54.282142 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:18:54.277747 4951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fdb2170-1155-429b-87e9-6827bdafbdee-maas-api-tls podName:0fdb2170-1155-429b-87e9-6827bdafbdee nodeName:}" failed. No retries permitted until 2026-10-30 07:18:54.777727196 +0000 UTC m=+1341.120775154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0fdb2170-1155-429b-87e9-6827bdafbdee-maas-api-tls") pod "maas-api-e2e-api-b-052254-bc7487fc6-qw9h7" (UID: "0fdb2170-1155-429b-87e9-6827bdafbdee") : secret "maas-api-e2e-api-b-052254-serving-cert" not found Oct 30 07:18:54.282142 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.278623 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fdb2170-1155-429b-87e9-6827bdafbdee-openshift-service-ca\") pod \"maas-api-e2e-api-b-052254-bc7487fc6-qw9h7\" (UID: \"0fdb2170-1155-429b-87e9-6827bdafbdee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-052254-bc7487fc6-qw9h7" Oct 30 07:18:54.284940 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.284914 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fdb2170-1155-429b-87e9-6827bdafbdee-metrics-certs\") pod \"maas-api-e2e-api-b-052254-bc7487fc6-qw9h7\" (UID: \"0fdb2170-1155-429b-87e9-6827bdafbdee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-052254-bc7487fc6-qw9h7" Oct 30 07:18:54.312776 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.312744 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm654\" (UniqueName: \"kubernetes.io/projected/0fdb2170-1155-429b-87e9-6827bdafbdee-kube-api-access-jm654\") pod \"maas-api-e2e-api-b-052254-bc7487fc6-qw9h7\" (UID: \"0fdb2170-1155-429b-87e9-6827bdafbdee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-052254-bc7487fc6-qw9h7" Oct 30 07:18:54.389997 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.389897 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-workload-socket\") pod \"e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:18:54.390287 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.390267 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2898e6e0-5229-4c67-b3e3-8f2076284b86-istio-token\") pod \"e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:18:54.390400 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.390385 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-credential-socket\") pod \"e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:18:54.390589 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.390558 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-istio-data\") pod \"e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:18:54.394594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.390719 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2898e6e0-5229-4c67-b3e3-8f2076284b86-istio-podinfo\") pod \"e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:18:54.394808 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.394786 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-istio-envoy\") pod \"e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:18:54.394958 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.394939 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h2x7\" (UniqueName: \"kubernetes.io/projected/2898e6e0-5229-4c67-b3e3-8f2076284b86-kube-api-access-5h2x7\") pod \"e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:18:54.395107 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.395090 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-workload-certs\") pod \"e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:18:54.395240 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.395223 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2898e6e0-5229-4c67-b3e3-8f2076284b86-istiod-ca-cert\") pod \"e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:18:54.529227 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.529176 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-istio-data\") pod \"e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:18:54.536109 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.536074 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2898e6e0-5229-4c67-b3e3-8f2076284b86-istio-podinfo\") pod \"e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:18:54.536910 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.536884 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-istio-envoy\") pod \"e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:18:54.537458 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.537429 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h2x7\" (UniqueName: \"kubernetes.io/projected/2898e6e0-5229-4c67-b3e3-8f2076284b86-kube-api-access-5h2x7\") pod \"e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:18:54.537653 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.535869 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-istio-data\") pod \"e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:18:54.537770 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.537755 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-workload-certs\") pod \"e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:18:54.537894 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.537878 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2898e6e0-5229-4c67-b3e3-8f2076284b86-istiod-ca-cert\") pod \"e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:18:54.538213 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.538193 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-workload-socket\") pod \"e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:18:54.545592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.538435 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2898e6e0-5229-4c67-b3e3-8f2076284b86-istio-token\") pod \"e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:18:54.545784 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.545765 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-credential-socket\") pod \"e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:18:54.546379 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.543105 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2898e6e0-5229-4c67-b3e3-8f2076284b86-istiod-ca-cert\") pod \"e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:18:54.546523 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.543322 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-workload-socket\") pod \"e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:18:54.546662 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.542182 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-workload-certs\") pod \"e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:18:54.546763 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.546124 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-credential-socket\") pod \"e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:18:54.566443 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.566389 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2898e6e0-5229-4c67-b3e3-8f2076284b86-istio-podinfo\") pod \"e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:18:54.572665 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.567560 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-istio-envoy\") pod \"e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:18:54.603649 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.601467 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h2x7\" (UniqueName: \"kubernetes.io/projected/2898e6e0-5229-4c67-b3e3-8f2076284b86-kube-api-access-5h2x7\") pod \"e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:18:54.603649 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.602100 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2898e6e0-5229-4c67-b3e3-8f2076284b86-istio-token\") pod \"e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:18:54.605885 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.605853 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:18:54.858930 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.858884 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fdb2170-1155-429b-87e9-6827bdafbdee-maas-api-tls\") pod \"maas-api-e2e-api-b-052254-bc7487fc6-qw9h7\" (UID: \"0fdb2170-1155-429b-87e9-6827bdafbdee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-052254-bc7487fc6-qw9h7" Oct 30 07:18:54.868650 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:54.868619 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fdb2170-1155-429b-87e9-6827bdafbdee-maas-api-tls\") pod \"maas-api-e2e-api-b-052254-bc7487fc6-qw9h7\" (UID: \"0fdb2170-1155-429b-87e9-6827bdafbdee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-052254-bc7487fc6-qw9h7" Oct 30 07:18:55.021000 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:55.019794 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-052254-bc7487fc6-qw9h7" Oct 30 07:18:55.141583 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:55.141074 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5"] Oct 30 07:18:55.142206 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:18:55.142155 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2898e6e0_5229_4c67_b3e3_8f2076284b86.slice/crio-be87df6dded6659fcb65ff40d21a5002f65e9b3bcbe5bb8a599fb6bdacb12652 WatchSource:0}: Error finding container be87df6dded6659fcb65ff40d21a5002f65e9b3bcbe5bb8a599fb6bdacb12652: Status 404 returned error can't find the container with id be87df6dded6659fcb65ff40d21a5002f65e9b3bcbe5bb8a599fb6bdacb12652 Oct 30 07:18:55.146300 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:55.146256 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:18:55.146876 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:55.146844 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:18:55.147057 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:55.147035 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:18:55.383608 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:55.379997 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-052254-bc7487fc6-qw9h7"] Oct 30 07:18:55.411330 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:18:55.394838 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fdb2170_1155_429b_87e9_6827bdafbdee.slice/crio-1024fd33d20ca81b62911d20e9e0ebb09c142f0cbc3677ec39a6d6ee3f40b0b3 WatchSource:0}: Error finding container 1024fd33d20ca81b62911d20e9e0ebb09c142f0cbc3677ec39a6d6ee3f40b0b3: Status 404 returned error can't find the container with id 1024fd33d20ca81b62911d20e9e0ebb09c142f0cbc3677ec39a6d6ee3f40b0b3 Oct 30 07:18:55.597858 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:55.597815 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" event={"ID":"2898e6e0-5229-4c67-b3e3-8f2076284b86","Type":"ContainerStarted","Data":"1e43449f9e6c7034fa598e65aef6052612b73a058c0388f7e0fd1239aa6afb21"} Oct 30 07:18:55.598028 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:55.597905 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" event={"ID":"2898e6e0-5229-4c67-b3e3-8f2076284b86","Type":"ContainerStarted","Data":"be87df6dded6659fcb65ff40d21a5002f65e9b3bcbe5bb8a599fb6bdacb12652"} Oct 30 07:18:55.600350 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:55.600321 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-052254-bc7487fc6-qw9h7" event={"ID":"0fdb2170-1155-429b-87e9-6827bdafbdee","Type":"ContainerStarted","Data":"1024fd33d20ca81b62911d20e9e0ebb09c142f0cbc3677ec39a6d6ee3f40b0b3"} Oct 30 07:18:55.606641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:55.606621 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:18:55.653131 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:55.652994 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" podStartSLOduration=1.6529784699999999 podStartE2EDuration="1.65297847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:18:55.651109259 +0000 UTC m=+1341.994157256" watchObservedRunningTime="2026-10-30 07:18:55.65297847 +0000 UTC m=+1341.996026477" Oct 30 07:18:56.556051 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:56.554401 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln"] Oct 30 07:18:56.556051 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:56.554668 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" podUID="1471e3cc-b331-4f73-aef3-33af56f4fed8" containerName="istio-proxy" containerID="cri-o://62e6f0675b9381df95248814c7e83e06ad2a7b1e9692e256faf9fc9cabfb1523" gracePeriod=30 Oct 30 07:18:56.608872 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:56.608839 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-052254-bc7487fc6-qw9h7" event={"ID":"0fdb2170-1155-429b-87e9-6827bdafbdee","Type":"ContainerStarted","Data":"b0fa91ca01bdb115c03247a3d4783559fca221fbaf870be90c32d0147e26e820"} Oct 30 07:18:56.609037 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:56.608931 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-052254-bc7487fc6-qw9h7" Oct 30 07:18:56.612298 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:56.612276 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:18:56.695597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:56.693908 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-052254-bc7487fc6-qw9h7" podStartSLOduration=2.2644431 podStartE2EDuration="2.693884464s" totalImagesPullingTime="429.441364ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:18:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:18:56.673318454 +0000 UTC m=+1343.016366420" watchObservedRunningTime="2026-10-30 07:18:56.693884464 +0000 UTC m=+1343.036932432" Oct 30 07:18:57.208104 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.208068 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp"] Oct 30 07:18:57.213382 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.213356 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:18:57.238173 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.238138 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp"] Oct 30 07:18:57.238449 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.238425 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-438855-openshift-default-dockercfg-478l6\"" Oct 30 07:18:57.275230 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.275199 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-workload-socket\") pod \"e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:18:57.275409 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.275237 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-workload-certs\") pod \"e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:18:57.275409 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.275271 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-istio-envoy\") pod \"e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:18:57.275524 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.275489 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/011a2449-9f58-427a-aa6e-89e4a911c2e5-istio-podinfo\") pod \"e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:18:57.275597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.275540 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-credential-socket\") pod \"e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:18:57.275664 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.275653 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-istio-data\") pod \"e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:18:57.275771 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.275756 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/011a2449-9f58-427a-aa6e-89e4a911c2e5-istio-token\") pod \"e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:18:57.275825 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.275783 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/011a2449-9f58-427a-aa6e-89e4a911c2e5-istiod-ca-cert\") pod \"e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:18:57.275877 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.275860 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pphq2\" (UniqueName: \"kubernetes.io/projected/011a2449-9f58-427a-aa6e-89e4a911c2e5-kube-api-access-pphq2\") pod \"e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:18:57.390596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.377612 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/011a2449-9f58-427a-aa6e-89e4a911c2e5-istio-token\") pod \"e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:18:57.390596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.377674 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/011a2449-9f58-427a-aa6e-89e4a911c2e5-istiod-ca-cert\") pod \"e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:18:57.390596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.377768 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pphq2\" (UniqueName: \"kubernetes.io/projected/011a2449-9f58-427a-aa6e-89e4a911c2e5-kube-api-access-pphq2\") pod \"e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:18:57.390596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.377941 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-workload-socket\") pod \"e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:18:57.390596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.377971 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-workload-certs\") pod \"e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:18:57.390596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.378013 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-istio-envoy\") pod \"e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:18:57.390596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.378164 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/011a2449-9f58-427a-aa6e-89e4a911c2e5-istio-podinfo\") pod \"e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:18:57.390596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.378255 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-credential-socket\") pod \"e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:18:57.390596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.378297 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-istio-data\") pod \"e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:18:57.390596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.378790 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-workload-socket\") pod \"e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:18:57.398592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.395838 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-istio-data\") pod \"e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:18:57.406598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.404556 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/011a2449-9f58-427a-aa6e-89e4a911c2e5-istiod-ca-cert\") pod \"e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:18:57.406598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.404752 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-credential-socket\") pod \"e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:18:57.406598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.404980 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-workload-certs\") pod \"e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:18:57.429663 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.426426 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-istio-envoy\") pod \"e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:18:57.429663 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.428843 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/011a2449-9f58-427a-aa6e-89e4a911c2e5-istio-token\") pod \"e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:18:57.439309 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.439201 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/011a2449-9f58-427a-aa6e-89e4a911c2e5-istio-podinfo\") pod \"e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:18:57.458635 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.454805 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pphq2\" (UniqueName: \"kubernetes.io/projected/011a2449-9f58-427a-aa6e-89e4a911c2e5-kube-api-access-pphq2\") pod \"e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:18:57.542826 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.542550 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:18:57.661480 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.660470 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:18:57.661970 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.661694 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:18:57.942945 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.941740 4951 patch_prober.go:29] interesting pod/e2e-api-a-318524-openshift-default-779d745bc5-ftbln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:18:57.942945 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.941807 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" podUID="1471e3cc-b331-4f73-aef3-33af56f4fed8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:18:57.986312 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.986285 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp"] Oct 30 07:18:57.987003 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:18:57.986970 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod011a2449_9f58_427a_aa6e_89e4a911c2e5.slice/crio-c770f3549bb3776cfaaa547af1a23dc1144820c172579953056f24642181a162 WatchSource:0}: Error finding container c770f3549bb3776cfaaa547af1a23dc1144820c172579953056f24642181a162: Status 404 returned error can't find the container with id c770f3549bb3776cfaaa547af1a23dc1144820c172579953056f24642181a162 Oct 30 07:18:57.990023 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.989986 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:18:57.990109 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.990063 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:18:57.990109 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:57.990093 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:18:58.673208 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:58.673171 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" event={"ID":"011a2449-9f58-427a-aa6e-89e4a911c2e5","Type":"ContainerStarted","Data":"b8f3461c9a0c2a6751c03305a15a997e9dc65e63ebf891040e7c6c33e984cf2a"} Oct 30 07:18:58.673809 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:58.673783 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" event={"ID":"011a2449-9f58-427a-aa6e-89e4a911c2e5","Type":"ContainerStarted","Data":"c770f3549bb3776cfaaa547af1a23dc1144820c172579953056f24642181a162"} Oct 30 07:18:58.735693 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:58.735632 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" podStartSLOduration=1.735614438 podStartE2EDuration="1.735614438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:18:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:18:58.732505297 +0000 UTC m=+1345.075553267" watchObservedRunningTime="2026-10-30 07:18:58.735614438 +0000 UTC m=+1345.078662403" Oct 30 07:18:59.543376 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:59.543337 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:18:59.549735 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:59.549696 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:18:59.681271 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:59.681240 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:18:59.682698 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:18:59.682673 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:19:00.074760 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:00.074723 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-cb54b9dbc-prwtd"] Oct 30 07:19:00.079268 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:00.079243 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-cb54b9dbc-prwtd" Oct 30 07:19:00.094597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:00.094560 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-7fdc99fcb6-bxqx7"] Oct 30 07:19:00.096172 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:00.096151 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0cd8f0-dockercfg-htn8r\"" Oct 30 07:19:00.096730 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:00.096698 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0cd8f0\"" Oct 30 07:19:00.099833 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:00.099816 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-7fdc99fcb6-bxqx7" Oct 30 07:19:00.102510 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:00.102482 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-cb54b9dbc-prwtd"] Oct 30 07:19:00.141022 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:00.140981 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-7fdc99fcb6-bxqx7"] Oct 30 07:19:00.290596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:00.290482 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed6e5a68-478f-4bad-8a26-1a9aecea9672-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0cd8f0-7fdc99fcb6-bxqx7\" (UID: \"ed6e5a68-478f-4bad-8a26-1a9aecea9672\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-7fdc99fcb6-bxqx7" Oct 30 07:19:00.290904 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:00.290653 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbjwc\" (UniqueName: \"kubernetes.io/projected/ed6e5a68-478f-4bad-8a26-1a9aecea9672-kube-api-access-vbjwc\") pod \"payload-processing-e2e-ipp-a-0cd8f0-7fdc99fcb6-bxqx7\" (UID: \"ed6e5a68-478f-4bad-8a26-1a9aecea9672\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-7fdc99fcb6-bxqx7" Oct 30 07:19:00.290904 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:00.290868 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730c927a-96d9-41ef-9e20-b44af912068e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0cd8f0-cb54b9dbc-prwtd\" (UID: \"730c927a-96d9-41ef-9e20-b44af912068e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-cb54b9dbc-prwtd" Oct 30 07:19:00.291029 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:00.290946 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qqcx\" (UniqueName: \"kubernetes.io/projected/730c927a-96d9-41ef-9e20-b44af912068e-kube-api-access-6qqcx\") pod \"payload-pre-processing-e2e-ipp-a-0cd8f0-cb54b9dbc-prwtd\" (UID: \"730c927a-96d9-41ef-9e20-b44af912068e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-cb54b9dbc-prwtd" Oct 30 07:19:00.441643 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:00.440530 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed6e5a68-478f-4bad-8a26-1a9aecea9672-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0cd8f0-7fdc99fcb6-bxqx7\" (UID: \"ed6e5a68-478f-4bad-8a26-1a9aecea9672\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-7fdc99fcb6-bxqx7" Oct 30 07:19:00.441643 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:00.440717 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbjwc\" (UniqueName: \"kubernetes.io/projected/ed6e5a68-478f-4bad-8a26-1a9aecea9672-kube-api-access-vbjwc\") pod \"payload-processing-e2e-ipp-a-0cd8f0-7fdc99fcb6-bxqx7\" (UID: \"ed6e5a68-478f-4bad-8a26-1a9aecea9672\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-7fdc99fcb6-bxqx7" Oct 30 07:19:00.441643 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:00.440901 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730c927a-96d9-41ef-9e20-b44af912068e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0cd8f0-cb54b9dbc-prwtd\" (UID: \"730c927a-96d9-41ef-9e20-b44af912068e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-cb54b9dbc-prwtd" Oct 30 07:19:00.441643 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:00.440972 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qqcx\" (UniqueName: \"kubernetes.io/projected/730c927a-96d9-41ef-9e20-b44af912068e-kube-api-access-6qqcx\") pod \"payload-pre-processing-e2e-ipp-a-0cd8f0-cb54b9dbc-prwtd\" (UID: \"730c927a-96d9-41ef-9e20-b44af912068e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-cb54b9dbc-prwtd" Oct 30 07:19:00.450853 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:00.442369 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed6e5a68-478f-4bad-8a26-1a9aecea9672-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0cd8f0-7fdc99fcb6-bxqx7\" (UID: \"ed6e5a68-478f-4bad-8a26-1a9aecea9672\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-7fdc99fcb6-bxqx7" Oct 30 07:19:00.450853 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:00.443464 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730c927a-96d9-41ef-9e20-b44af912068e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0cd8f0-cb54b9dbc-prwtd\" (UID: \"730c927a-96d9-41ef-9e20-b44af912068e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-cb54b9dbc-prwtd" Oct 30 07:19:00.514601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:00.500189 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qqcx\" (UniqueName: \"kubernetes.io/projected/730c927a-96d9-41ef-9e20-b44af912068e-kube-api-access-6qqcx\") pod \"payload-pre-processing-e2e-ipp-a-0cd8f0-cb54b9dbc-prwtd\" (UID: \"730c927a-96d9-41ef-9e20-b44af912068e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-cb54b9dbc-prwtd" Oct 30 07:19:00.514601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:00.514110 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbjwc\" (UniqueName: \"kubernetes.io/projected/ed6e5a68-478f-4bad-8a26-1a9aecea9672-kube-api-access-vbjwc\") pod \"payload-processing-e2e-ipp-a-0cd8f0-7fdc99fcb6-bxqx7\" (UID: \"ed6e5a68-478f-4bad-8a26-1a9aecea9672\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-7fdc99fcb6-bxqx7" Oct 30 07:19:00.695797 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:00.695710 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-cb54b9dbc-prwtd" Oct 30 07:19:00.717797 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:00.717767 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-7fdc99fcb6-bxqx7" Oct 30 07:19:01.003812 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:01.003781 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-cb54b9dbc-prwtd"] Oct 30 07:19:01.004119 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:19:01.004086 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod730c927a_96d9_41ef_9e20_b44af912068e.slice/crio-e26fa592090f0a0c56b6a14b870ba4ef08f33b4a40b252fd7b15a3db1f9923be WatchSource:0}: Error finding container e26fa592090f0a0c56b6a14b870ba4ef08f33b4a40b252fd7b15a3db1f9923be: Status 404 returned error can't find the container with id e26fa592090f0a0c56b6a14b870ba4ef08f33b4a40b252fd7b15a3db1f9923be Oct 30 07:19:01.041218 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:19:01.041193 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded6e5a68_478f_4bad_8a26_1a9aecea9672.slice/crio-561f1b3ebcdb405b6bc81941d7923ef06e23c2b244f21f617fbbbeb14c488730 WatchSource:0}: Error finding container 561f1b3ebcdb405b6bc81941d7923ef06e23c2b244f21f617fbbbeb14c488730: Status 404 returned error can't find the container with id 561f1b3ebcdb405b6bc81941d7923ef06e23c2b244f21f617fbbbeb14c488730 Oct 30 07:19:01.041408 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:01.041387 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-7fdc99fcb6-bxqx7"] Oct 30 07:19:01.697636 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:01.697555 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-cb54b9dbc-prwtd" event={"ID":"730c927a-96d9-41ef-9e20-b44af912068e","Type":"ContainerStarted","Data":"74be6be526959c4f2073cf8b40002e347a9c4f53ee41b3f18f7c364fda794027"} Oct 30 07:19:01.698082 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:01.697692 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-cb54b9dbc-prwtd" event={"ID":"730c927a-96d9-41ef-9e20-b44af912068e","Type":"ContainerStarted","Data":"e26fa592090f0a0c56b6a14b870ba4ef08f33b4a40b252fd7b15a3db1f9923be"} Oct 30 07:19:01.698082 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:01.697987 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-cb54b9dbc-prwtd" Oct 30 07:19:01.700171 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:01.700136 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-7fdc99fcb6-bxqx7" event={"ID":"ed6e5a68-478f-4bad-8a26-1a9aecea9672","Type":"ContainerStarted","Data":"5c55073ad4c961162068b47bb7461a107d76af92d7e455988ec548ddef2bcde7"} Oct 30 07:19:01.700329 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:01.700191 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-7fdc99fcb6-bxqx7" event={"ID":"ed6e5a68-478f-4bad-8a26-1a9aecea9672","Type":"ContainerStarted","Data":"561f1b3ebcdb405b6bc81941d7923ef06e23c2b244f21f617fbbbeb14c488730"} Oct 30 07:19:01.700329 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:01.700218 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-7fdc99fcb6-bxqx7" Oct 30 07:19:01.702775 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:01.702748 4951 generic.go:356] "Generic (PLEG): container finished" podID="1471e3cc-b331-4f73-aef3-33af56f4fed8" containerID="62e6f0675b9381df95248814c7e83e06ad2a7b1e9692e256faf9fc9cabfb1523" exitCode=0 Oct 30 07:19:01.703641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:01.703619 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" event={"ID":"1471e3cc-b331-4f73-aef3-33af56f4fed8","Type":"ContainerDied","Data":"62e6f0675b9381df95248814c7e83e06ad2a7b1e9692e256faf9fc9cabfb1523"} Oct 30 07:19:01.703725 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:01.703681 4951 scope.go:122] "RemoveContainer" containerID="04252430b67c2e0fcb118208575eb0e0566410813ce043a880bed4b65d6372ea" Oct 30 07:19:01.741285 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:01.741232 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-cb54b9dbc-prwtd" podStartSLOduration=2.741215911 podStartE2EDuration="2.741215911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:19:01.737351653 +0000 UTC m=+1348.080399647" watchObservedRunningTime="2026-10-30 07:19:01.741215911 +0000 UTC m=+1348.084263920" Oct 30 07:19:01.789385 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:01.789332 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-7fdc99fcb6-bxqx7" podStartSLOduration=1.7893181230000001 podStartE2EDuration="1.789318123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:19:01.787282535 +0000 UTC m=+1348.130330518" watchObservedRunningTime="2026-10-30 07:19:01.789318123 +0000 UTC m=+1348.132366090" Oct 30 07:19:01.869024 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:01.868996 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:19:01.987220 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:01.987176 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-workload-socket\") pod \"1471e3cc-b331-4f73-aef3-33af56f4fed8\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " Oct 30 07:19:01.987565 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:01.987510 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-istio-envoy\") pod \"1471e3cc-b331-4f73-aef3-33af56f4fed8\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " Oct 30 07:19:01.987807 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:01.987682 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-workload-certs\") pod \"1471e3cc-b331-4f73-aef3-33af56f4fed8\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " Oct 30 07:19:01.987807 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:01.987731 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1471e3cc-b331-4f73-aef3-33af56f4fed8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1471e3cc-b331-4f73-aef3-33af56f4fed8-istiod-ca-cert\") pod \"1471e3cc-b331-4f73-aef3-33af56f4fed8\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " Oct 30 07:19:01.987953 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:01.987874 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1471e3cc-b331-4f73-aef3-33af56f4fed8-istio-token\" (UniqueName: \"kubernetes.io/projected/1471e3cc-b331-4f73-aef3-33af56f4fed8-istio-token\") pod \"1471e3cc-b331-4f73-aef3-33af56f4fed8\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " Oct 30 07:19:01.988010 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:01.987951 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1471e3cc-b331-4f73-aef3-33af56f4fed8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1471e3cc-b331-4f73-aef3-33af56f4fed8-istio-podinfo\") pod \"1471e3cc-b331-4f73-aef3-33af56f4fed8\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " Oct 30 07:19:01.988076 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:01.988056 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1471e3cc-b331-4f73-aef3-33af56f4fed8-kube-api-access-nlqlc\" (UniqueName: \"kubernetes.io/projected/1471e3cc-b331-4f73-aef3-33af56f4fed8-kube-api-access-nlqlc\") pod \"1471e3cc-b331-4f73-aef3-33af56f4fed8\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " Oct 30 07:19:01.988245 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:01.988131 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-credential-socket\") pod \"1471e3cc-b331-4f73-aef3-33af56f4fed8\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " Oct 30 07:19:01.988245 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:01.988200 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-istio-data\") pod \"1471e3cc-b331-4f73-aef3-33af56f4fed8\" (UID: \"1471e3cc-b331-4f73-aef3-33af56f4fed8\") " Oct 30 07:19:01.993949 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:01.993391 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1471e3cc-b331-4f73-aef3-33af56f4fed8-istiod-ca-cert" pod "1471e3cc-b331-4f73-aef3-33af56f4fed8" (UID: "1471e3cc-b331-4f73-aef3-33af56f4fed8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:19:01.993949 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:01.993745 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-istio-data" pod "1471e3cc-b331-4f73-aef3-33af56f4fed8" (UID: "1471e3cc-b331-4f73-aef3-33af56f4fed8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:02.006738 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:02.006695 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-workload-socket" pod "1471e3cc-b331-4f73-aef3-33af56f4fed8" (UID: "1471e3cc-b331-4f73-aef3-33af56f4fed8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:02.013550 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:02.013511 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-credential-socket" pod "1471e3cc-b331-4f73-aef3-33af56f4fed8" (UID: "1471e3cc-b331-4f73-aef3-33af56f4fed8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:02.014936 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:02.014885 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-workload-certs" pod "1471e3cc-b331-4f73-aef3-33af56f4fed8" (UID: "1471e3cc-b331-4f73-aef3-33af56f4fed8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:02.018004 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:02.017957 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-istio-envoy" pod "1471e3cc-b331-4f73-aef3-33af56f4fed8" (UID: "1471e3cc-b331-4f73-aef3-33af56f4fed8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:02.020136 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:02.020088 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1471e3cc-b331-4f73-aef3-33af56f4fed8-istio-token" pod "1471e3cc-b331-4f73-aef3-33af56f4fed8" (UID: "1471e3cc-b331-4f73-aef3-33af56f4fed8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:02.020318 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:02.020173 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1471e3cc-b331-4f73-aef3-33af56f4fed8-kube-api-access-nlqlc" pod "1471e3cc-b331-4f73-aef3-33af56f4fed8" (UID: "1471e3cc-b331-4f73-aef3-33af56f4fed8"). InnerVolumeSpecName "kube-api-access-nlqlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:02.020855 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:02.020827 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1471e3cc-b331-4f73-aef3-33af56f4fed8-istio-podinfo" pod "1471e3cc-b331-4f73-aef3-33af56f4fed8" (UID: "1471e3cc-b331-4f73-aef3-33af56f4fed8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:19:02.090478 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:02.090392 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:02.090478 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:02.090420 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:02.090478 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:02.090430 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:02.090478 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:02.090440 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1471e3cc-b331-4f73-aef3-33af56f4fed8-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:02.090478 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:02.090450 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1471e3cc-b331-4f73-aef3-33af56f4fed8-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:02.090478 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:02.090458 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1471e3cc-b331-4f73-aef3-33af56f4fed8-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:02.090478 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:02.090467 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlqlc\" (UniqueName: \"kubernetes.io/projected/1471e3cc-b331-4f73-aef3-33af56f4fed8-kube-api-access-nlqlc\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:02.090478 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:02.090478 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:02.090478 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:02.090486 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1471e3cc-b331-4f73-aef3-33af56f4fed8-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:02.666037 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:02.666007 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-052254-bc7487fc6-qw9h7" Oct 30 07:19:02.712431 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:02.712400 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" Oct 30 07:19:02.712900 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:02.712425 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln" event={"ID":"1471e3cc-b331-4f73-aef3-33af56f4fed8","Type":"ContainerDied","Data":"c1ab3de76dc2b4e54b275f7e6f487dea121167252d6fff791d7fc321e2c7957e"} Oct 30 07:19:02.712900 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:02.712488 4951 scope.go:122] "RemoveContainer" containerID="62e6f0675b9381df95248814c7e83e06ad2a7b1e9692e256faf9fc9cabfb1523" Oct 30 07:19:02.770705 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:02.770660 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln"] Oct 30 07:19:02.774039 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:02.774012 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-318524-openshift-default-779d745bc5-ftbln"] Oct 30 07:19:03.088457 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:03.088376 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-7c7f65f99-ljj28"] Oct 30 07:19:03.089103 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:03.089089 4951 cpu_manager.go:414] "containerMap: removing container" podUID="1471e3cc-b331-4f73-aef3-33af56f4fed8" containerName="istio-proxy" Oct 30 07:19:03.089161 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:03.089103 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1471e3cc-b331-4f73-aef3-33af56f4fed8" containerName="istio-proxy" Oct 30 07:19:03.089161 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:03.089122 4951 cpu_manager.go:414] "containerMap: removing container" podUID="1471e3cc-b331-4f73-aef3-33af56f4fed8" containerName="istio-proxy" Oct 30 07:19:03.089161 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:03.089128 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1471e3cc-b331-4f73-aef3-33af56f4fed8" containerName="istio-proxy" Oct 30 07:19:03.089276 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:03.089265 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="1471e3cc-b331-4f73-aef3-33af56f4fed8" containerName="istio-proxy" Oct 30 07:19:03.089310 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:03.089277 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="1471e3cc-b331-4f73-aef3-33af56f4fed8" containerName="istio-proxy" Oct 30 07:19:03.094413 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:03.094394 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-7c7f65f99-ljj28" Oct 30 07:19:03.109036 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:03.109012 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-438855\"" Oct 30 07:19:03.109198 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:03.109065 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-438855-dockercfg-j475v\"" Oct 30 07:19:03.113827 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:03.113800 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-7c7f65f99-ljj28"] Oct 30 07:19:03.321598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:03.318288 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-438855-f9597c45d-wl76r"] Oct 30 07:19:03.338646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:03.337082 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e0f9017-d26f-4aff-99ef-c07825f692ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-438855-7c7f65f99-ljj28\" (UID: \"7e0f9017-d26f-4aff-99ef-c07825f692ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-7c7f65f99-ljj28" Oct 30 07:19:03.338646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:03.337150 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm4nq\" (UniqueName: \"kubernetes.io/projected/7e0f9017-d26f-4aff-99ef-c07825f692ad-kube-api-access-rm4nq\") pod \"payload-pre-processing-e2e-ipp-a-438855-7c7f65f99-ljj28\" (UID: \"7e0f9017-d26f-4aff-99ef-c07825f692ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-7c7f65f99-ljj28" Oct 30 07:19:03.358594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:03.355996 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-f9597c45d-wl76r" Oct 30 07:19:03.437981 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:03.430442 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-438855-f9597c45d-wl76r"] Oct 30 07:19:03.483814 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:03.476078 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e0f9017-d26f-4aff-99ef-c07825f692ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-438855-7c7f65f99-ljj28\" (UID: \"7e0f9017-d26f-4aff-99ef-c07825f692ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-7c7f65f99-ljj28" Oct 30 07:19:03.483814 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:03.476133 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm4nq\" (UniqueName: \"kubernetes.io/projected/7e0f9017-d26f-4aff-99ef-c07825f692ad-kube-api-access-rm4nq\") pod \"payload-pre-processing-e2e-ipp-a-438855-7c7f65f99-ljj28\" (UID: \"7e0f9017-d26f-4aff-99ef-c07825f692ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-7c7f65f99-ljj28" Oct 30 07:19:03.483814 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:03.478225 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e0f9017-d26f-4aff-99ef-c07825f692ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-438855-7c7f65f99-ljj28\" (UID: \"7e0f9017-d26f-4aff-99ef-c07825f692ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-7c7f65f99-ljj28" Oct 30 07:19:03.513590 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:03.513506 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm4nq\" (UniqueName: \"kubernetes.io/projected/7e0f9017-d26f-4aff-99ef-c07825f692ad-kube-api-access-rm4nq\") pod \"payload-pre-processing-e2e-ipp-a-438855-7c7f65f99-ljj28\" (UID: \"7e0f9017-d26f-4aff-99ef-c07825f692ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-7c7f65f99-ljj28" Oct 30 07:19:03.578211 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:03.578151 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cdqb\" (UniqueName: \"kubernetes.io/projected/126b094f-2a13-4eff-bd0b-3c0e2a52853e-kube-api-access-6cdqb\") pod \"payload-processing-e2e-ipp-a-438855-f9597c45d-wl76r\" (UID: \"126b094f-2a13-4eff-bd0b-3c0e2a52853e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-f9597c45d-wl76r" Oct 30 07:19:03.578211 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:03.578216 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/126b094f-2a13-4eff-bd0b-3c0e2a52853e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-438855-f9597c45d-wl76r\" (UID: \"126b094f-2a13-4eff-bd0b-3c0e2a52853e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-f9597c45d-wl76r" Oct 30 07:19:03.679615 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:03.679496 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cdqb\" (UniqueName: \"kubernetes.io/projected/126b094f-2a13-4eff-bd0b-3c0e2a52853e-kube-api-access-6cdqb\") pod \"payload-processing-e2e-ipp-a-438855-f9597c45d-wl76r\" (UID: \"126b094f-2a13-4eff-bd0b-3c0e2a52853e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-f9597c45d-wl76r" Oct 30 07:19:03.679615 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:03.679537 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/126b094f-2a13-4eff-bd0b-3c0e2a52853e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-438855-f9597c45d-wl76r\" (UID: \"126b094f-2a13-4eff-bd0b-3c0e2a52853e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-f9597c45d-wl76r" Oct 30 07:19:03.680311 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:03.680294 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/126b094f-2a13-4eff-bd0b-3c0e2a52853e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-438855-f9597c45d-wl76r\" (UID: \"126b094f-2a13-4eff-bd0b-3c0e2a52853e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-f9597c45d-wl76r" Oct 30 07:19:03.703497 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:03.703466 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cdqb\" (UniqueName: \"kubernetes.io/projected/126b094f-2a13-4eff-bd0b-3c0e2a52853e-kube-api-access-6cdqb\") pod \"payload-processing-e2e-ipp-a-438855-f9597c45d-wl76r\" (UID: \"126b094f-2a13-4eff-bd0b-3c0e2a52853e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-f9597c45d-wl76r" Oct 30 07:19:03.709176 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:03.709157 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-7c7f65f99-ljj28" Oct 30 07:19:03.786612 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:03.780749 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1471e3cc-b331-4f73-aef3-33af56f4fed8" path="/var/lib/kubelet/pods/1471e3cc-b331-4f73-aef3-33af56f4fed8/volumes" Oct 30 07:19:03.815261 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:03.815228 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-f9597c45d-wl76r" Oct 30 07:19:04.031981 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:04.031928 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-7c7f65f99-ljj28"] Oct 30 07:19:04.038184 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:19:04.038140 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0f9017_d26f_4aff_99ef_c07825f692ad.slice/crio-445a54bc40a324ebc410088a6e5c1e4b5fca2d0f49fa6eeef73e01f3842938bd WatchSource:0}: Error finding container 445a54bc40a324ebc410088a6e5c1e4b5fca2d0f49fa6eeef73e01f3842938bd: Status 404 returned error can't find the container with id 445a54bc40a324ebc410088a6e5c1e4b5fca2d0f49fa6eeef73e01f3842938bd Oct 30 07:19:04.240173 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:19:04.240098 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod126b094f_2a13_4eff_bd0b_3c0e2a52853e.slice/crio-c76c4d7021d8964ca227f0ab3628c3fa4ae30f52c4dc67149b5c2cb28a7ce731 WatchSource:0}: Error finding container c76c4d7021d8964ca227f0ab3628c3fa4ae30f52c4dc67149b5c2cb28a7ce731: Status 404 returned error can't find the container with id c76c4d7021d8964ca227f0ab3628c3fa4ae30f52c4dc67149b5c2cb28a7ce731 Oct 30 07:19:04.241308 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:04.241281 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-438855-f9597c45d-wl76r"] Oct 30 07:19:04.733041 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:04.733002 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-7c7f65f99-ljj28" event={"ID":"7e0f9017-d26f-4aff-99ef-c07825f692ad","Type":"ContainerStarted","Data":"8c079531fdff293e0dca53e80a982c0d2acd45f76c958129254bcb0b6b95d51b"} Oct 30 07:19:04.733271 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:04.733069 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-7c7f65f99-ljj28" event={"ID":"7e0f9017-d26f-4aff-99ef-c07825f692ad","Type":"ContainerStarted","Data":"445a54bc40a324ebc410088a6e5c1e4b5fca2d0f49fa6eeef73e01f3842938bd"} Oct 30 07:19:04.733271 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:04.733121 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-7c7f65f99-ljj28" Oct 30 07:19:04.735036 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:04.735003 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-f9597c45d-wl76r" event={"ID":"126b094f-2a13-4eff-bd0b-3c0e2a52853e","Type":"ContainerStarted","Data":"613de3349c9c44ab5950b257b9e773099875cf7c35ac9f3963d2e864f179ddb2"} Oct 30 07:19:04.735180 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:04.735046 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-f9597c45d-wl76r" event={"ID":"126b094f-2a13-4eff-bd0b-3c0e2a52853e","Type":"ContainerStarted","Data":"c76c4d7021d8964ca227f0ab3628c3fa4ae30f52c4dc67149b5c2cb28a7ce731"} Oct 30 07:19:04.735180 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:04.735159 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-f9597c45d-wl76r" Oct 30 07:19:04.777147 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:04.777063 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-7c7f65f99-ljj28" podStartSLOduration=1.777041087 podStartE2EDuration="1.777041087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:19:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:19:04.773034074 +0000 UTC m=+1351.116082040" watchObservedRunningTime="2026-10-30 07:19:04.777041087 +0000 UTC m=+1351.120089053" Oct 30 07:19:04.818786 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:04.818722 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-f9597c45d-wl76r" podStartSLOduration=1.818705233 podStartE2EDuration="1.818705233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:19:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:19:04.814710217 +0000 UTC m=+1351.157758183" watchObservedRunningTime="2026-10-30 07:19:04.818705233 +0000 UTC m=+1351.161753199" Oct 30 07:19:12.715775 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:12.715743 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-7fdc99fcb6-bxqx7" Oct 30 07:19:12.716191 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:12.715816 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-cb54b9dbc-prwtd" Oct 30 07:19:15.467160 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:15.467119 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn"] Oct 30 07:19:15.467736 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:15.467394 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" podUID="89d560f3-70af-44f7-8084-4ca608698965" containerName="istio-proxy" containerID="cri-o://190c321dd69ed8191538ac789bc0e5da75bdf68196e27387c762dd73f8184c76" gracePeriod=30 Oct 30 07:19:15.480950 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:15.480915 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577"] Oct 30 07:19:15.481209 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:15.481181 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" podUID="5c9605f5-3aa1-4f48-8ec6-1b709863952f" containerName="istio-proxy" containerID="cri-o://1040c97709b625ce4ea3ceb35d515970bc078dc34ac185cb993bff353c1eb58f" gracePeriod=30 Oct 30 07:19:15.488481 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:15.488446 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r"] Oct 30 07:19:15.488755 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:15.488728 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" podUID="d159ec3e-d629-4a66-b0c7-17d00455c088" containerName="istio-proxy" containerID="cri-o://048063b3f194485da85239b7e2ab0b91ec2139d12b8119e9b5f195b3884714eb" gracePeriod=30 Oct 30 07:19:15.495031 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:15.494995 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk"] Oct 30 07:19:15.495262 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:15.495213 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" podUID="98e03afc-4edf-499e-a63a-0f92372c65a9" containerName="istio-proxy" containerID="cri-o://7d2ee23c69b3c2cd6bfed0b70f9dbe24fbf20a11062dc61c7a36a682ce783262" gracePeriod=30 Oct 30 07:19:15.720369 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:15.720272 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4"] Oct 30 07:19:15.766675 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:15.766640 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4" Oct 30 07:19:15.783440 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:15.783405 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4"] Oct 30 07:19:15.783637 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:15.783501 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-7c7f65f99-ljj28" Oct 30 07:19:15.783637 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:15.783551 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-f9597c45d-wl76r" Oct 30 07:19:15.787644 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:15.787622 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-0cd8f0-serving-cert\"" Oct 30 07:19:15.796797 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:15.796775 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4f5cb00-0e55-4cc7-8599-1b6d5d94de02-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4\" (UID: \"e4f5cb00-0e55-4cc7-8599-1b6d5d94de02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4" Oct 30 07:19:15.797163 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:15.797126 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4f5cb00-0e55-4cc7-8599-1b6d5d94de02-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4\" (UID: \"e4f5cb00-0e55-4cc7-8599-1b6d5d94de02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4" Oct 30 07:19:15.798345 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:15.797312 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pmct\" (UniqueName: \"kubernetes.io/projected/e4f5cb00-0e55-4cc7-8599-1b6d5d94de02-kube-api-access-8pmct\") pod \"maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4\" (UID: \"e4f5cb00-0e55-4cc7-8599-1b6d5d94de02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4" Oct 30 07:19:15.798345 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:15.797763 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4f5cb00-0e55-4cc7-8599-1b6d5d94de02-metrics-certs\") pod \"maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4\" (UID: \"e4f5cb00-0e55-4cc7-8599-1b6d5d94de02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4" Oct 30 07:19:15.898784 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:15.898752 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4f5cb00-0e55-4cc7-8599-1b6d5d94de02-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4\" (UID: \"e4f5cb00-0e55-4cc7-8599-1b6d5d94de02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4" Oct 30 07:19:15.898993 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:15.898824 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4f5cb00-0e55-4cc7-8599-1b6d5d94de02-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4\" (UID: \"e4f5cb00-0e55-4cc7-8599-1b6d5d94de02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4" Oct 30 07:19:15.898993 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:15.898902 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pmct\" (UniqueName: \"kubernetes.io/projected/e4f5cb00-0e55-4cc7-8599-1b6d5d94de02-kube-api-access-8pmct\") pod \"maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4\" (UID: \"e4f5cb00-0e55-4cc7-8599-1b6d5d94de02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4" Oct 30 07:19:15.898993 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:15.898957 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4f5cb00-0e55-4cc7-8599-1b6d5d94de02-metrics-certs\") pod \"maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4\" (UID: \"e4f5cb00-0e55-4cc7-8599-1b6d5d94de02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4" Oct 30 07:19:15.899968 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:15.899943 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4f5cb00-0e55-4cc7-8599-1b6d5d94de02-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4\" (UID: \"e4f5cb00-0e55-4cc7-8599-1b6d5d94de02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4" Oct 30 07:19:15.902479 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:15.902452 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4f5cb00-0e55-4cc7-8599-1b6d5d94de02-metrics-certs\") pod \"maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4\" (UID: \"e4f5cb00-0e55-4cc7-8599-1b6d5d94de02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4" Oct 30 07:19:15.903305 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:15.903280 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4f5cb00-0e55-4cc7-8599-1b6d5d94de02-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4\" (UID: \"e4f5cb00-0e55-4cc7-8599-1b6d5d94de02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4" Oct 30 07:19:15.986142 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:15.986069 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pmct\" (UniqueName: \"kubernetes.io/projected/e4f5cb00-0e55-4cc7-8599-1b6d5d94de02-kube-api-access-8pmct\") pod \"maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4\" (UID: \"e4f5cb00-0e55-4cc7-8599-1b6d5d94de02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4" Oct 30 07:19:16.092151 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:16.091862 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4" Oct 30 07:19:16.767618 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:16.766553 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4"] Oct 30 07:19:17.172633 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:17.172543 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4" event={"ID":"e4f5cb00-0e55-4cc7-8599-1b6d5d94de02","Type":"ContainerStarted","Data":"c7fa307ac884221ea3feb0029c28b711c2fba2a1f74c4c1bf8ae7f6a5f06571c"} Oct 30 07:19:18.183006 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:18.182962 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4" event={"ID":"e4f5cb00-0e55-4cc7-8599-1b6d5d94de02","Type":"ContainerStarted","Data":"ac131475081d2d18791ad863c39547f311df749c0fe11cf8f6acea2e5a3f69a3"} Oct 30 07:19:18.183499 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:18.183070 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4" Oct 30 07:19:18.259048 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:18.258985 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4" podStartSLOduration=2.728645922 podStartE2EDuration="3.258966892s" totalImagesPullingTime="530.32097ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:19:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:19:18.256481742 +0000 UTC m=+1364.599529708" watchObservedRunningTime="2026-10-30 07:19:18.258966892 +0000 UTC m=+1364.602014858" Oct 30 07:19:19.575218 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:19.575177 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2"] Oct 30 07:19:19.581232 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:19.581207 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2" Oct 30 07:19:19.604130 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:19.604094 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2"] Oct 30 07:19:19.607013 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:19.606817 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-438855-serving-cert\"" Oct 30 07:19:19.673979 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:19.673943 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgl4b\" (UniqueName: \"kubernetes.io/projected/258478b4-7854-48a9-8138-707475c24418-kube-api-access-dgl4b\") pod \"maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2\" (UID: \"258478b4-7854-48a9-8138-707475c24418\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2" Oct 30 07:19:19.674149 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:19.674009 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/258478b4-7854-48a9-8138-707475c24418-metrics-certs\") pod \"maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2\" (UID: \"258478b4-7854-48a9-8138-707475c24418\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2" Oct 30 07:19:19.674244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:19.674217 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/258478b4-7854-48a9-8138-707475c24418-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2\" (UID: \"258478b4-7854-48a9-8138-707475c24418\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2" Oct 30 07:19:19.674374 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:19.674354 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/258478b4-7854-48a9-8138-707475c24418-maas-api-tls\") pod \"maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2\" (UID: \"258478b4-7854-48a9-8138-707475c24418\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2" Oct 30 07:19:19.776209 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:19.776169 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/258478b4-7854-48a9-8138-707475c24418-maas-api-tls\") pod \"maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2\" (UID: \"258478b4-7854-48a9-8138-707475c24418\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2" Oct 30 07:19:19.776453 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:19.776396 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgl4b\" (UniqueName: \"kubernetes.io/projected/258478b4-7854-48a9-8138-707475c24418-kube-api-access-dgl4b\") pod \"maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2\" (UID: \"258478b4-7854-48a9-8138-707475c24418\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2" Oct 30 07:19:19.776565 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:19.776477 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/258478b4-7854-48a9-8138-707475c24418-metrics-certs\") pod \"maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2\" (UID: \"258478b4-7854-48a9-8138-707475c24418\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2" Oct 30 07:19:19.776824 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:19.776800 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/258478b4-7854-48a9-8138-707475c24418-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2\" (UID: \"258478b4-7854-48a9-8138-707475c24418\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2" Oct 30 07:19:19.777658 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:19.777631 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/258478b4-7854-48a9-8138-707475c24418-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2\" (UID: \"258478b4-7854-48a9-8138-707475c24418\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2" Oct 30 07:19:19.779245 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:19.779220 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/258478b4-7854-48a9-8138-707475c24418-metrics-certs\") pod \"maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2\" (UID: \"258478b4-7854-48a9-8138-707475c24418\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2" Oct 30 07:19:19.779414 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:19.779315 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/258478b4-7854-48a9-8138-707475c24418-maas-api-tls\") pod \"maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2\" (UID: \"258478b4-7854-48a9-8138-707475c24418\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2" Oct 30 07:19:19.800100 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:19.800065 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgl4b\" (UniqueName: \"kubernetes.io/projected/258478b4-7854-48a9-8138-707475c24418-kube-api-access-dgl4b\") pod \"maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2\" (UID: \"258478b4-7854-48a9-8138-707475c24418\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2" Oct 30 07:19:19.911649 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:19.904858 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2" Oct 30 07:19:20.291537 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:20.291499 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2"] Oct 30 07:19:20.292774 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:19:20.292739 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod258478b4_7854_48a9_8138_707475c24418.slice/crio-89a57af86a95eff36b7f1ae7c1cbba259798a0f45b1f0a84a3d2d8e29ae1cab4 WatchSource:0}: Error finding container 89a57af86a95eff36b7f1ae7c1cbba259798a0f45b1f0a84a3d2d8e29ae1cab4: Status 404 returned error can't find the container with id 89a57af86a95eff36b7f1ae7c1cbba259798a0f45b1f0a84a3d2d8e29ae1cab4 Oct 30 07:19:20.472719 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:20.472673 4951 patch_prober.go:29] interesting pod/e2e-api-b-052254-openshift-default-57b897c995-9hsd2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:19:20.472883 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:20.472741 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" podUID="94280cc1-4ffe-4862-9cfc-1d61949679ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:19:21.066440 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.065696 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:19:21.141332 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.140056 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-credential-socket\") pod \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " Oct 30 07:19:21.141332 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.140132 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-workload-socket\") pod \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " Oct 30 07:19:21.141332 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.140220 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istio-token\" (UniqueName: \"kubernetes.io/projected/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istio-token\") pod \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " Oct 30 07:19:21.141332 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.140268 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istiod-ca-cert\") pod \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " Oct 30 07:19:21.141332 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.140342 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-workload-certs\") pod \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " Oct 30 07:19:21.141332 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.140479 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c9605f5-3aa1-4f48-8ec6-1b709863952f-kube-api-access-5dq2c\" (UniqueName: \"kubernetes.io/projected/5c9605f5-3aa1-4f48-8ec6-1b709863952f-kube-api-access-5dq2c\") pod \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " Oct 30 07:19:21.141332 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.140520 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istio-podinfo\") pod \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " Oct 30 07:19:21.141332 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.140548 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istio-data\") pod \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " Oct 30 07:19:21.143122 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.142020 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istio-envoy\") pod \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\" (UID: \"5c9605f5-3aa1-4f48-8ec6-1b709863952f\") " Oct 30 07:19:21.152259 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.145078 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-workload-socket" pod "5c9605f5-3aa1-4f48-8ec6-1b709863952f" (UID: "5c9605f5-3aa1-4f48-8ec6-1b709863952f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:21.152259 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.145455 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-credential-socket" pod "5c9605f5-3aa1-4f48-8ec6-1b709863952f" (UID: "5c9605f5-3aa1-4f48-8ec6-1b709863952f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:21.152259 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.150607 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istio-data" pod "5c9605f5-3aa1-4f48-8ec6-1b709863952f" (UID: "5c9605f5-3aa1-4f48-8ec6-1b709863952f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:21.152259 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.152023 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-workload-certs" pod "5c9605f5-3aa1-4f48-8ec6-1b709863952f" (UID: "5c9605f5-3aa1-4f48-8ec6-1b709863952f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:21.152259 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.152220 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istiod-ca-cert" pod "5c9605f5-3aa1-4f48-8ec6-1b709863952f" (UID: "5c9605f5-3aa1-4f48-8ec6-1b709863952f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:19:21.164445 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.164308 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istio-token" pod "5c9605f5-3aa1-4f48-8ec6-1b709863952f" (UID: "5c9605f5-3aa1-4f48-8ec6-1b709863952f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:21.166725 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.166625 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c9605f5-3aa1-4f48-8ec6-1b709863952f-kube-api-access-5dq2c" pod "5c9605f5-3aa1-4f48-8ec6-1b709863952f" (UID: "5c9605f5-3aa1-4f48-8ec6-1b709863952f"). InnerVolumeSpecName "kube-api-access-5dq2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:21.168424 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.168026 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istio-podinfo" pod "5c9605f5-3aa1-4f48-8ec6-1b709863952f" (UID: "5c9605f5-3aa1-4f48-8ec6-1b709863952f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:19:21.172662 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.172625 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istio-envoy" pod "5c9605f5-3aa1-4f48-8ec6-1b709863952f" (UID: "5c9605f5-3aa1-4f48-8ec6-1b709863952f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:21.174892 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.174869 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:19:21.180949 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.180749 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:19:21.187630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.187434 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:19:21.218192 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.218090 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2" event={"ID":"258478b4-7854-48a9-8138-707475c24418","Type":"ContainerStarted","Data":"5f3967617cc02bc0e3b794513de12da7ec894e06e3564beb7957b6336f6f10b2"} Oct 30 07:19:21.218192 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.218165 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2" event={"ID":"258478b4-7854-48a9-8138-707475c24418","Type":"ContainerStarted","Data":"89a57af86a95eff36b7f1ae7c1cbba259798a0f45b1f0a84a3d2d8e29ae1cab4"} Oct 30 07:19:21.218506 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.218440 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2" Oct 30 07:19:21.220026 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.219990 4951 generic.go:356] "Generic (PLEG): container finished" podID="98e03afc-4edf-499e-a63a-0f92372c65a9" containerID="7d2ee23c69b3c2cd6bfed0b70f9dbe24fbf20a11062dc61c7a36a682ce783262" exitCode=0 Oct 30 07:19:21.220152 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.220043 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" Oct 30 07:19:21.220152 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.220068 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" event={"ID":"98e03afc-4edf-499e-a63a-0f92372c65a9","Type":"ContainerDied","Data":"7d2ee23c69b3c2cd6bfed0b70f9dbe24fbf20a11062dc61c7a36a682ce783262"} Oct 30 07:19:21.220152 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.220125 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk" event={"ID":"98e03afc-4edf-499e-a63a-0f92372c65a9","Type":"ContainerDied","Data":"06d6d2deaf77aca3ff0e5dbe185311cdfceaa744aaa521a8ed675d6a5d3ecce4"} Oct 30 07:19:21.220318 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.220161 4951 scope.go:122] "RemoveContainer" containerID="7d2ee23c69b3c2cd6bfed0b70f9dbe24fbf20a11062dc61c7a36a682ce783262" Oct 30 07:19:21.223079 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.222684 4951 generic.go:356] "Generic (PLEG): container finished" podID="89d560f3-70af-44f7-8084-4ca608698965" containerID="190c321dd69ed8191538ac789bc0e5da75bdf68196e27387c762dd73f8184c76" exitCode=0 Oct 30 07:19:21.223079 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.222803 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" event={"ID":"89d560f3-70af-44f7-8084-4ca608698965","Type":"ContainerDied","Data":"190c321dd69ed8191538ac789bc0e5da75bdf68196e27387c762dd73f8184c76"} Oct 30 07:19:21.223079 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.222844 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" event={"ID":"89d560f3-70af-44f7-8084-4ca608698965","Type":"ContainerDied","Data":"929ba8187bc7004209cd9b283b13dea040a50ab802a3457bf162be7b2a4242df"} Oct 30 07:19:21.223079 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.222922 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" Oct 30 07:19:21.226922 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.226725 4951 generic.go:356] "Generic (PLEG): container finished" podID="d159ec3e-d629-4a66-b0c7-17d00455c088" containerID="048063b3f194485da85239b7e2ab0b91ec2139d12b8119e9b5f195b3884714eb" exitCode=0 Oct 30 07:19:21.226922 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.226793 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" Oct 30 07:19:21.226922 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.226843 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" event={"ID":"d159ec3e-d629-4a66-b0c7-17d00455c088","Type":"ContainerDied","Data":"048063b3f194485da85239b7e2ab0b91ec2139d12b8119e9b5f195b3884714eb"} Oct 30 07:19:21.226922 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.226880 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r" event={"ID":"d159ec3e-d629-4a66-b0c7-17d00455c088","Type":"ContainerDied","Data":"5609c3aefda890e80c98e07f622113f8bc266680fe9177cb25dc15d177e1b8b3"} Oct 30 07:19:21.233596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.229832 4951 generic.go:356] "Generic (PLEG): container finished" podID="5c9605f5-3aa1-4f48-8ec6-1b709863952f" containerID="1040c97709b625ce4ea3ceb35d515970bc078dc34ac185cb993bff353c1eb58f" exitCode=0 Oct 30 07:19:21.233596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.229947 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" event={"ID":"5c9605f5-3aa1-4f48-8ec6-1b709863952f","Type":"ContainerDied","Data":"1040c97709b625ce4ea3ceb35d515970bc078dc34ac185cb993bff353c1eb58f"} Oct 30 07:19:21.233596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.229983 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" event={"ID":"5c9605f5-3aa1-4f48-8ec6-1b709863952f","Type":"ContainerDied","Data":"ecff96ffaa1444a2e50773d0c10217e3f3806376ca37e78c0e43a20163ed05da"} Oct 30 07:19:21.233596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.230046 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577" Oct 30 07:19:21.244757 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.244726 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98e03afc-4edf-499e-a63a-0f92372c65a9-istio-token\" (UniqueName: \"kubernetes.io/projected/98e03afc-4edf-499e-a63a-0f92372c65a9-istio-token\") pod \"98e03afc-4edf-499e-a63a-0f92372c65a9\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " Oct 30 07:19:21.245473 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.245434 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-workload-socket\") pod \"98e03afc-4edf-499e-a63a-0f92372c65a9\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " Oct 30 07:19:21.246009 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.245989 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-workload-certs\") pod \"d159ec3e-d629-4a66-b0c7-17d00455c088\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " Oct 30 07:19:21.246395 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.246377 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d159ec3e-d629-4a66-b0c7-17d00455c088-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d159ec3e-d629-4a66-b0c7-17d00455c088-istio-podinfo\") pod \"d159ec3e-d629-4a66-b0c7-17d00455c088\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " Oct 30 07:19:21.246980 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.245921 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-workload-socket" pod "98e03afc-4edf-499e-a63a-0f92372c65a9" (UID: "98e03afc-4edf-499e-a63a-0f92372c65a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:21.246980 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.246330 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-workload-certs" pod "d159ec3e-d629-4a66-b0c7-17d00455c088" (UID: "d159ec3e-d629-4a66-b0c7-17d00455c088"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:21.247437 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.246919 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-istio-envoy\") pod \"89d560f3-70af-44f7-8084-4ca608698965\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " Oct 30 07:19:21.247624 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.247605 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89d560f3-70af-44f7-8084-4ca608698965-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89d560f3-70af-44f7-8084-4ca608698965-istio-podinfo\") pod \"89d560f3-70af-44f7-8084-4ca608698965\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " Oct 30 07:19:21.248101 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.248083 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98e03afc-4edf-499e-a63a-0f92372c65a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98e03afc-4edf-499e-a63a-0f92372c65a9-istio-podinfo\") pod \"98e03afc-4edf-499e-a63a-0f92372c65a9\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " Oct 30 07:19:21.248320 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.248303 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-istio-envoy\") pod \"98e03afc-4edf-499e-a63a-0f92372c65a9\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " Oct 30 07:19:21.248754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.248738 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-credential-socket\") pod \"89d560f3-70af-44f7-8084-4ca608698965\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " Oct 30 07:19:21.248890 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.248876 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-istio-data\") pod \"d159ec3e-d629-4a66-b0c7-17d00455c088\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " Oct 30 07:19:21.249005 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.248992 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d159ec3e-d629-4a66-b0c7-17d00455c088-istio-token\" (UniqueName: \"kubernetes.io/projected/d159ec3e-d629-4a66-b0c7-17d00455c088-istio-token\") pod \"d159ec3e-d629-4a66-b0c7-17d00455c088\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " Oct 30 07:19:21.249102 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.249090 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-workload-socket\") pod \"d159ec3e-d629-4a66-b0c7-17d00455c088\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " Oct 30 07:19:21.249242 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.249228 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-credential-socket\") pod \"d159ec3e-d629-4a66-b0c7-17d00455c088\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " Oct 30 07:19:21.249356 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.249343 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98e03afc-4edf-499e-a63a-0f92372c65a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98e03afc-4edf-499e-a63a-0f92372c65a9-istiod-ca-cert\") pod \"98e03afc-4edf-499e-a63a-0f92372c65a9\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " Oct 30 07:19:21.249504 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.249485 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-istio-data\") pod \"89d560f3-70af-44f7-8084-4ca608698965\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " Oct 30 07:19:21.249823 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.249767 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-workload-certs\") pod \"89d560f3-70af-44f7-8084-4ca608698965\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " Oct 30 07:19:21.250025 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.250008 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-workload-socket\") pod \"89d560f3-70af-44f7-8084-4ca608698965\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " Oct 30 07:19:21.250164 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.250148 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d159ec3e-d629-4a66-b0c7-17d00455c088-kube-api-access-gl4jd\" (UniqueName: \"kubernetes.io/projected/d159ec3e-d629-4a66-b0c7-17d00455c088-kube-api-access-gl4jd\") pod \"d159ec3e-d629-4a66-b0c7-17d00455c088\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " Oct 30 07:19:21.250313 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.250298 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d159ec3e-d629-4a66-b0c7-17d00455c088-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d159ec3e-d629-4a66-b0c7-17d00455c088-istiod-ca-cert\") pod \"d159ec3e-d629-4a66-b0c7-17d00455c088\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " Oct 30 07:19:21.250482 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.250465 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89d560f3-70af-44f7-8084-4ca608698965-istio-token\" (UniqueName: \"kubernetes.io/projected/89d560f3-70af-44f7-8084-4ca608698965-istio-token\") pod \"89d560f3-70af-44f7-8084-4ca608698965\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " Oct 30 07:19:21.250628 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.250611 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-istio-envoy\") pod \"d159ec3e-d629-4a66-b0c7-17d00455c088\" (UID: \"d159ec3e-d629-4a66-b0c7-17d00455c088\") " Oct 30 07:19:21.250766 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.250750 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-credential-socket\") pod \"98e03afc-4edf-499e-a63a-0f92372c65a9\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " Oct 30 07:19:21.250894 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.250880 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-workload-certs\") pod \"98e03afc-4edf-499e-a63a-0f92372c65a9\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " Oct 30 07:19:21.251056 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.251028 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-istio-data\") pod \"98e03afc-4edf-499e-a63a-0f92372c65a9\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " Oct 30 07:19:21.251265 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.251248 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89d560f3-70af-44f7-8084-4ca608698965-kube-api-access-7pp2v\" (UniqueName: \"kubernetes.io/projected/89d560f3-70af-44f7-8084-4ca608698965-kube-api-access-7pp2v\") pod \"89d560f3-70af-44f7-8084-4ca608698965\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " Oct 30 07:19:21.251413 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.251392 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89d560f3-70af-44f7-8084-4ca608698965-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89d560f3-70af-44f7-8084-4ca608698965-istiod-ca-cert\") pod \"89d560f3-70af-44f7-8084-4ca608698965\" (UID: \"89d560f3-70af-44f7-8084-4ca608698965\") " Oct 30 07:19:21.251584 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.251550 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98e03afc-4edf-499e-a63a-0f92372c65a9-kube-api-access-hfg4r\" (UniqueName: \"kubernetes.io/projected/98e03afc-4edf-499e-a63a-0f92372c65a9-kube-api-access-hfg4r\") pod \"98e03afc-4edf-499e-a63a-0f92372c65a9\" (UID: \"98e03afc-4edf-499e-a63a-0f92372c65a9\") " Oct 30 07:19:21.254122 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.252991 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-istio-envoy" pod "98e03afc-4edf-499e-a63a-0f92372c65a9" (UID: "98e03afc-4edf-499e-a63a-0f92372c65a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:21.254122 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.253347 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d159ec3e-d629-4a66-b0c7-17d00455c088-istiod-ca-cert" pod "d159ec3e-d629-4a66-b0c7-17d00455c088" (UID: "d159ec3e-d629-4a66-b0c7-17d00455c088"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:19:21.254122 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.253656 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-credential-socket" pod "89d560f3-70af-44f7-8084-4ca608698965" (UID: "89d560f3-70af-44f7-8084-4ca608698965"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:21.254122 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.253841 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-istio-data" pod "d159ec3e-d629-4a66-b0c7-17d00455c088" (UID: "d159ec3e-d629-4a66-b0c7-17d00455c088"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:21.254389 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.254358 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98e03afc-4edf-499e-a63a-0f92372c65a9-istio-podinfo" pod "98e03afc-4edf-499e-a63a-0f92372c65a9" (UID: "98e03afc-4edf-499e-a63a-0f92372c65a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:19:21.255109 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.254621 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-istio-data" pod "98e03afc-4edf-499e-a63a-0f92372c65a9" (UID: "98e03afc-4edf-499e-a63a-0f92372c65a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:21.255109 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.254647 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-istio-envoy" pod "89d560f3-70af-44f7-8084-4ca608698965" (UID: "89d560f3-70af-44f7-8084-4ca608698965"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:21.255109 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.254866 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-credential-socket" pod "98e03afc-4edf-499e-a63a-0f92372c65a9" (UID: "98e03afc-4edf-499e-a63a-0f92372c65a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:21.256307 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.256278 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d159ec3e-d629-4a66-b0c7-17d00455c088-istio-token" pod "d159ec3e-d629-4a66-b0c7-17d00455c088" (UID: "d159ec3e-d629-4a66-b0c7-17d00455c088"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:21.256528 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.256505 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-workload-socket" pod "d159ec3e-d629-4a66-b0c7-17d00455c088" (UID: "d159ec3e-d629-4a66-b0c7-17d00455c088"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:21.256839 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.256815 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-credential-socket" pod "d159ec3e-d629-4a66-b0c7-17d00455c088" (UID: "d159ec3e-d629-4a66-b0c7-17d00455c088"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.256948 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-istio-envoy" pod "d159ec3e-d629-4a66-b0c7-17d00455c088" (UID: "d159ec3e-d629-4a66-b0c7-17d00455c088"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.257126 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98e03afc-4edf-499e-a63a-0f92372c65a9-istiod-ca-cert" pod "98e03afc-4edf-499e-a63a-0f92372c65a9" (UID: "98e03afc-4edf-499e-a63a-0f92372c65a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.257364 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-istio-data" pod "89d560f3-70af-44f7-8084-4ca608698965" (UID: "89d560f3-70af-44f7-8084-4ca608698965"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.257528 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.257551 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.257552 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-workload-certs" pod "89d560f3-70af-44f7-8084-4ca608698965" (UID: "89d560f3-70af-44f7-8084-4ca608698965"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.257585 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.257601 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.257615 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.257628 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.257641 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98e03afc-4edf-499e-a63a-0f92372c65a9-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.257655 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.257667 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.257679 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.257692 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.257705 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.257720 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d159ec3e-d629-4a66-b0c7-17d00455c088-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.257734 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.257746 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.257761 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-workload-socket" pod "89d560f3-70af-44f7-8084-4ca608698965" (UID: "89d560f3-70af-44f7-8084-4ca608698965"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.257762 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98e03afc-4edf-499e-a63a-0f92372c65a9-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.257799 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dq2c\" (UniqueName: \"kubernetes.io/projected/5c9605f5-3aa1-4f48-8ec6-1b709863952f-kube-api-access-5dq2c\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.257818 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.257836 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.257852 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d159ec3e-d629-4a66-b0c7-17d00455c088-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.257865 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c9605f5-3aa1-4f48-8ec6-1b709863952f-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.257877 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d159ec3e-d629-4a66-b0c7-17d00455c088-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.257891 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.257903 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.258122 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89d560f3-70af-44f7-8084-4ca608698965-istio-token" pod "89d560f3-70af-44f7-8084-4ca608698965" (UID: "89d560f3-70af-44f7-8084-4ca608698965"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.258203 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-workload-certs" pod "98e03afc-4edf-499e-a63a-0f92372c65a9" (UID: "98e03afc-4edf-499e-a63a-0f92372c65a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.258502 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89d560f3-70af-44f7-8084-4ca608698965-istiod-ca-cert" pod "89d560f3-70af-44f7-8084-4ca608698965" (UID: "89d560f3-70af-44f7-8084-4ca608698965"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.260000 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98e03afc-4edf-499e-a63a-0f92372c65a9-kube-api-access-hfg4r" pod "98e03afc-4edf-499e-a63a-0f92372c65a9" (UID: "98e03afc-4edf-499e-a63a-0f92372c65a9"). InnerVolumeSpecName "kube-api-access-hfg4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:21.274754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.260195 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d159ec3e-d629-4a66-b0c7-17d00455c088-kube-api-access-gl4jd" pod "d159ec3e-d629-4a66-b0c7-17d00455c088" (UID: "d159ec3e-d629-4a66-b0c7-17d00455c088"). InnerVolumeSpecName "kube-api-access-gl4jd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:21.288927 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.282887 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98e03afc-4edf-499e-a63a-0f92372c65a9-istio-token" pod "98e03afc-4edf-499e-a63a-0f92372c65a9" (UID: "98e03afc-4edf-499e-a63a-0f92372c65a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:21.288927 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.283888 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89d560f3-70af-44f7-8084-4ca608698965-istio-podinfo" pod "89d560f3-70af-44f7-8084-4ca608698965" (UID: "89d560f3-70af-44f7-8084-4ca608698965"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:19:21.288927 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.284171 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d159ec3e-d629-4a66-b0c7-17d00455c088-istio-podinfo" pod "d159ec3e-d629-4a66-b0c7-17d00455c088" (UID: "d159ec3e-d629-4a66-b0c7-17d00455c088"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:19:21.288927 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.287306 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89d560f3-70af-44f7-8084-4ca608698965-kube-api-access-7pp2v" pod "89d560f3-70af-44f7-8084-4ca608698965" (UID: "89d560f3-70af-44f7-8084-4ca608698965"). InnerVolumeSpecName "kube-api-access-7pp2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:21.306600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.306544 4951 scope.go:122] "RemoveContainer" containerID="7d2ee23c69b3c2cd6bfed0b70f9dbe24fbf20a11062dc61c7a36a682ce783262" Oct 30 07:19:21.307407 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:19:21.307091 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d2ee23c69b3c2cd6bfed0b70f9dbe24fbf20a11062dc61c7a36a682ce783262\": container with ID starting with 7d2ee23c69b3c2cd6bfed0b70f9dbe24fbf20a11062dc61c7a36a682ce783262 not found: ID does not exist" containerID="7d2ee23c69b3c2cd6bfed0b70f9dbe24fbf20a11062dc61c7a36a682ce783262" Oct 30 07:19:21.307407 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.307120 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d2ee23c69b3c2cd6bfed0b70f9dbe24fbf20a11062dc61c7a36a682ce783262"} err="failed to get container status \"7d2ee23c69b3c2cd6bfed0b70f9dbe24fbf20a11062dc61c7a36a682ce783262\": rpc error: code = NotFound desc = could not find container \"7d2ee23c69b3c2cd6bfed0b70f9dbe24fbf20a11062dc61c7a36a682ce783262\": container with ID starting with 7d2ee23c69b3c2cd6bfed0b70f9dbe24fbf20a11062dc61c7a36a682ce783262 not found: ID does not exist" Oct 30 07:19:21.307407 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.307143 4951 scope.go:122] "RemoveContainer" containerID="190c321dd69ed8191538ac789bc0e5da75bdf68196e27387c762dd73f8184c76" Oct 30 07:19:21.336389 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.333501 4951 scope.go:122] "RemoveContainer" containerID="190c321dd69ed8191538ac789bc0e5da75bdf68196e27387c762dd73f8184c76" Oct 30 07:19:21.336389 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:19:21.335429 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"190c321dd69ed8191538ac789bc0e5da75bdf68196e27387c762dd73f8184c76\": container with ID starting with 190c321dd69ed8191538ac789bc0e5da75bdf68196e27387c762dd73f8184c76 not found: ID does not exist" containerID="190c321dd69ed8191538ac789bc0e5da75bdf68196e27387c762dd73f8184c76" Oct 30 07:19:21.336389 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.335466 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"190c321dd69ed8191538ac789bc0e5da75bdf68196e27387c762dd73f8184c76"} err="failed to get container status \"190c321dd69ed8191538ac789bc0e5da75bdf68196e27387c762dd73f8184c76\": rpc error: code = NotFound desc = could not find container \"190c321dd69ed8191538ac789bc0e5da75bdf68196e27387c762dd73f8184c76\": container with ID starting with 190c321dd69ed8191538ac789bc0e5da75bdf68196e27387c762dd73f8184c76 not found: ID does not exist" Oct 30 07:19:21.336389 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.335489 4951 scope.go:122] "RemoveContainer" containerID="048063b3f194485da85239b7e2ab0b91ec2139d12b8119e9b5f195b3884714eb" Oct 30 07:19:21.366388 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.359680 4951 scope.go:122] "RemoveContainer" containerID="048063b3f194485da85239b7e2ab0b91ec2139d12b8119e9b5f195b3884714eb" Oct 30 07:19:21.366388 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:19:21.360488 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"048063b3f194485da85239b7e2ab0b91ec2139d12b8119e9b5f195b3884714eb\": container with ID starting with 048063b3f194485da85239b7e2ab0b91ec2139d12b8119e9b5f195b3884714eb not found: ID does not exist" containerID="048063b3f194485da85239b7e2ab0b91ec2139d12b8119e9b5f195b3884714eb" Oct 30 07:19:21.366388 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.360521 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"048063b3f194485da85239b7e2ab0b91ec2139d12b8119e9b5f195b3884714eb"} err="failed to get container status \"048063b3f194485da85239b7e2ab0b91ec2139d12b8119e9b5f195b3884714eb\": rpc error: code = NotFound desc = could not find container \"048063b3f194485da85239b7e2ab0b91ec2139d12b8119e9b5f195b3884714eb\": container with ID starting with 048063b3f194485da85239b7e2ab0b91ec2139d12b8119e9b5f195b3884714eb not found: ID does not exist" Oct 30 07:19:21.366388 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.360545 4951 scope.go:122] "RemoveContainer" containerID="1040c97709b625ce4ea3ceb35d515970bc078dc34ac185cb993bff353c1eb58f" Oct 30 07:19:21.366388 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.360867 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfg4r\" (UniqueName: \"kubernetes.io/projected/98e03afc-4edf-499e-a63a-0f92372c65a9-kube-api-access-hfg4r\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.366388 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.360892 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98e03afc-4edf-499e-a63a-0f92372c65a9-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.366388 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.360908 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d159ec3e-d629-4a66-b0c7-17d00455c088-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.366388 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.360922 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89d560f3-70af-44f7-8084-4ca608698965-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.366388 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.360934 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.366388 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.360946 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.366388 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.360957 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d560f3-70af-44f7-8084-4ca608698965-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.366388 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.360970 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl4jd\" (UniqueName: \"kubernetes.io/projected/d159ec3e-d629-4a66-b0c7-17d00455c088-kube-api-access-gl4jd\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.366388 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.360983 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89d560f3-70af-44f7-8084-4ca608698965-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.366388 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.360995 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98e03afc-4edf-499e-a63a-0f92372c65a9-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.366388 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.361007 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pp2v\" (UniqueName: \"kubernetes.io/projected/89d560f3-70af-44f7-8084-4ca608698965-kube-api-access-7pp2v\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.366388 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.361019 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89d560f3-70af-44f7-8084-4ca608698965-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.370645 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.370046 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2" podStartSLOduration=1.991663592 podStartE2EDuration="2.370030947s" totalImagesPullingTime="378.367355ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:19:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:19:21.361162138 +0000 UTC m=+1367.704210104" watchObservedRunningTime="2026-10-30 07:19:21.370030947 +0000 UTC m=+1367.713078911" Oct 30 07:19:21.391696 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.391674 4951 scope.go:122] "RemoveContainer" containerID="1040c97709b625ce4ea3ceb35d515970bc078dc34ac185cb993bff353c1eb58f" Oct 30 07:19:21.392129 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:19:21.392096 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1040c97709b625ce4ea3ceb35d515970bc078dc34ac185cb993bff353c1eb58f\": container with ID starting with 1040c97709b625ce4ea3ceb35d515970bc078dc34ac185cb993bff353c1eb58f not found: ID does not exist" containerID="1040c97709b625ce4ea3ceb35d515970bc078dc34ac185cb993bff353c1eb58f" Oct 30 07:19:21.392250 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.392140 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1040c97709b625ce4ea3ceb35d515970bc078dc34ac185cb993bff353c1eb58f"} err="failed to get container status \"1040c97709b625ce4ea3ceb35d515970bc078dc34ac185cb993bff353c1eb58f\": rpc error: code = NotFound desc = could not find container \"1040c97709b625ce4ea3ceb35d515970bc078dc34ac185cb993bff353c1eb58f\": container with ID starting with 1040c97709b625ce4ea3ceb35d515970bc078dc34ac185cb993bff353c1eb58f not found: ID does not exist" Oct 30 07:19:21.456726 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.456683 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577"] Oct 30 07:19:21.460674 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.460635 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-sd577"] Oct 30 07:19:21.622153 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.622120 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk"] Oct 30 07:19:21.628701 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.628675 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-7v9wk"] Oct 30 07:19:21.662734 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.662704 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r"] Oct 30 07:19:21.669494 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.669460 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-t4p2r"] Oct 30 07:19:21.708546 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.708514 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn"] Oct 30 07:19:21.712006 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.711983 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn"] Oct 30 07:19:21.778496 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.778455 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c9605f5-3aa1-4f48-8ec6-1b709863952f" path="/var/lib/kubelet/pods/5c9605f5-3aa1-4f48-8ec6-1b709863952f/volumes" Oct 30 07:19:21.779300 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.779276 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89d560f3-70af-44f7-8084-4ca608698965" path="/var/lib/kubelet/pods/89d560f3-70af-44f7-8084-4ca608698965/volumes" Oct 30 07:19:21.779966 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.779940 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98e03afc-4edf-499e-a63a-0f92372c65a9" path="/var/lib/kubelet/pods/98e03afc-4edf-499e-a63a-0f92372c65a9/volumes" Oct 30 07:19:21.781095 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:21.781069 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d159ec3e-d629-4a66-b0c7-17d00455c088" path="/var/lib/kubelet/pods/d159ec3e-d629-4a66-b0c7-17d00455c088/volumes" Oct 30 07:19:22.163762 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:22.163716 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:19:22.164165 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:22.163784 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-lrkbn" podUID="89d560f3-70af-44f7-8084-4ca608698965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:19:23.025294 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:19:23.025259 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:19:23.267412 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:23.267380 4951 generic.go:356] "Generic (PLEG): container finished" podID="94280cc1-4ffe-4862-9cfc-1d61949679ed" containerID="f10083df04ab335b726acd7c22e316dd8cabbf4aabd50511ecd7911f40f4c291" exitCode=0 Oct 30 07:19:23.267908 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:23.267460 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" event={"ID":"94280cc1-4ffe-4862-9cfc-1d61949679ed","Type":"ContainerDied","Data":"f10083df04ab335b726acd7c22e316dd8cabbf4aabd50511ecd7911f40f4c291"} Oct 30 07:19:23.267908 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:23.267897 4951 scope.go:122] "RemoveContainer" containerID="f10083df04ab335b726acd7c22e316dd8cabbf4aabd50511ecd7911f40f4c291" Oct 30 07:19:23.269398 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:23.269369 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:19:23.269481 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:23.269460 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:19:23.269649 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:23.269507 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:19:24.199397 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:24.199367 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4" Oct 30 07:19:24.282408 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:24.282368 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" event={"ID":"94280cc1-4ffe-4862-9cfc-1d61949679ed","Type":"ContainerStarted","Data":"99b7e757f1676994d8b02862bbf67ac99c2a1a2d418d81dfaf8f51e991653ec1"} Oct 30 07:19:24.992600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:24.992510 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:19:24.992833 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:24.992796 4951 patch_prober.go:29] interesting pod/e2e-api-b-052254-openshift-default-57b897c995-9hsd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 07:19:24.992917 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:24.992886 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" podUID="94280cc1-4ffe-4862-9cfc-1d61949679ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 07:19:25.992557 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:25.992518 4951 patch_prober.go:29] interesting pod/e2e-api-b-052254-openshift-default-57b897c995-9hsd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 07:19:25.993186 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:25.992601 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" podUID="94280cc1-4ffe-4862-9cfc-1d61949679ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 07:19:26.993341 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:26.993302 4951 patch_prober.go:29] interesting pod/e2e-api-b-052254-openshift-default-57b897c995-9hsd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 07:19:26.993841 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:26.993360 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" podUID="94280cc1-4ffe-4862-9cfc-1d61949679ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 07:19:27.255667 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:27.255632 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2" Oct 30 07:19:27.762504 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:27.762465 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp"] Oct 30 07:19:27.763092 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:27.763054 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp" podUID="a8b8448b-c2f2-4e27-bea9-a5d4cd44466f" containerName="maas-api" containerID="cri-o://7ccfc7fa0d5e661db5a9921c332bc3688a43e9637f756eef37b4a71e1b7f8037" gracePeriod=30 Oct 30 07:19:27.993132 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:27.993087 4951 patch_prober.go:29] interesting pod/e2e-api-b-052254-openshift-default-57b897c995-9hsd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 07:19:27.993314 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:27.993146 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" podUID="94280cc1-4ffe-4862-9cfc-1d61949679ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 07:19:28.175980 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:28.175873 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp" Oct 30 07:19:28.288316 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:28.288285 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8b8448b-c2f2-4e27-bea9-a5d4cd44466f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8b8448b-c2f2-4e27-bea9-a5d4cd44466f-maas-api-tls\") pod \"a8b8448b-c2f2-4e27-bea9-a5d4cd44466f\" (UID: \"a8b8448b-c2f2-4e27-bea9-a5d4cd44466f\") " Oct 30 07:19:28.288622 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:28.288604 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8b8448b-c2f2-4e27-bea9-a5d4cd44466f-kube-api-access-vbzm5\" (UniqueName: \"kubernetes.io/projected/a8b8448b-c2f2-4e27-bea9-a5d4cd44466f-kube-api-access-vbzm5\") pod \"a8b8448b-c2f2-4e27-bea9-a5d4cd44466f\" (UID: \"a8b8448b-c2f2-4e27-bea9-a5d4cd44466f\") " Oct 30 07:19:28.288893 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:28.288873 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8b8448b-c2f2-4e27-bea9-a5d4cd44466f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8b8448b-c2f2-4e27-bea9-a5d4cd44466f-metrics-certs\") pod \"a8b8448b-c2f2-4e27-bea9-a5d4cd44466f\" (UID: \"a8b8448b-c2f2-4e27-bea9-a5d4cd44466f\") " Oct 30 07:19:28.289162 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:28.289143 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8b8448b-c2f2-4e27-bea9-a5d4cd44466f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8b8448b-c2f2-4e27-bea9-a5d4cd44466f-openshift-service-ca\") pod \"a8b8448b-c2f2-4e27-bea9-a5d4cd44466f\" (UID: \"a8b8448b-c2f2-4e27-bea9-a5d4cd44466f\") " Oct 30 07:19:28.289456 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:28.289434 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8b8448b-c2f2-4e27-bea9-a5d4cd44466f-openshift-service-ca" pod "a8b8448b-c2f2-4e27-bea9-a5d4cd44466f" (UID: "a8b8448b-c2f2-4e27-bea9-a5d4cd44466f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:19:28.289863 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:28.289848 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8b8448b-c2f2-4e27-bea9-a5d4cd44466f-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:28.291266 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:28.291234 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8b8448b-c2f2-4e27-bea9-a5d4cd44466f-maas-api-tls" pod "a8b8448b-c2f2-4e27-bea9-a5d4cd44466f" (UID: "a8b8448b-c2f2-4e27-bea9-a5d4cd44466f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:19:28.291918 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:28.291895 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8b8448b-c2f2-4e27-bea9-a5d4cd44466f-metrics-certs" pod "a8b8448b-c2f2-4e27-bea9-a5d4cd44466f" (UID: "a8b8448b-c2f2-4e27-bea9-a5d4cd44466f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:19:28.292100 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:28.292075 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b8448b-c2f2-4e27-bea9-a5d4cd44466f-kube-api-access-vbzm5" pod "a8b8448b-c2f2-4e27-bea9-a5d4cd44466f" (UID: "a8b8448b-c2f2-4e27-bea9-a5d4cd44466f"). InnerVolumeSpecName "kube-api-access-vbzm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:28.367418 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:28.367322 4951 generic.go:356] "Generic (PLEG): container finished" podID="a8b8448b-c2f2-4e27-bea9-a5d4cd44466f" containerID="7ccfc7fa0d5e661db5a9921c332bc3688a43e9637f756eef37b4a71e1b7f8037" exitCode=0 Oct 30 07:19:28.369066 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:28.367462 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp" event={"ID":"a8b8448b-c2f2-4e27-bea9-a5d4cd44466f","Type":"ContainerDied","Data":"7ccfc7fa0d5e661db5a9921c332bc3688a43e9637f756eef37b4a71e1b7f8037"} Oct 30 07:19:28.369066 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:28.367753 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp" event={"ID":"a8b8448b-c2f2-4e27-bea9-a5d4cd44466f","Type":"ContainerDied","Data":"f0ad5e9e56761c075dbf8bcb80c5d73234b1ac2f5dcecb2d6852bc258c92cdc2"} Oct 30 07:19:28.369066 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:28.367791 4951 scope.go:122] "RemoveContainer" containerID="7ccfc7fa0d5e661db5a9921c332bc3688a43e9637f756eef37b4a71e1b7f8037" Oct 30 07:19:28.369066 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:28.367546 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp" Oct 30 07:19:28.382149 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:28.382125 4951 scope.go:122] "RemoveContainer" containerID="7ccfc7fa0d5e661db5a9921c332bc3688a43e9637f756eef37b4a71e1b7f8037" Oct 30 07:19:28.382535 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:19:28.382511 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ccfc7fa0d5e661db5a9921c332bc3688a43e9637f756eef37b4a71e1b7f8037\": container with ID starting with 7ccfc7fa0d5e661db5a9921c332bc3688a43e9637f756eef37b4a71e1b7f8037 not found: ID does not exist" containerID="7ccfc7fa0d5e661db5a9921c332bc3688a43e9637f756eef37b4a71e1b7f8037" Oct 30 07:19:28.382604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:28.382545 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ccfc7fa0d5e661db5a9921c332bc3688a43e9637f756eef37b4a71e1b7f8037"} err="failed to get container status \"7ccfc7fa0d5e661db5a9921c332bc3688a43e9637f756eef37b4a71e1b7f8037\": rpc error: code = NotFound desc = could not find container \"7ccfc7fa0d5e661db5a9921c332bc3688a43e9637f756eef37b4a71e1b7f8037\": container with ID starting with 7ccfc7fa0d5e661db5a9921c332bc3688a43e9637f756eef37b4a71e1b7f8037 not found: ID does not exist" Oct 30 07:19:28.391489 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:28.391453 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbzm5\" (UniqueName: \"kubernetes.io/projected/a8b8448b-c2f2-4e27-bea9-a5d4cd44466f-kube-api-access-vbzm5\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:28.391489 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:28.391487 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8b8448b-c2f2-4e27-bea9-a5d4cd44466f-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:28.391489 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:28.391498 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8b8448b-c2f2-4e27-bea9-a5d4cd44466f-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:28.471708 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:28.471629 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp"] Oct 30 07:19:28.488597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:28.484230 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6cc5f2-7976946c46-z24jp"] Oct 30 07:19:28.993363 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:28.993331 4951 patch_prober.go:29] interesting pod/e2e-api-b-052254-openshift-default-57b897c995-9hsd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 07:19:28.993536 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:28.993384 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" podUID="94280cc1-4ffe-4862-9cfc-1d61949679ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 07:19:29.776365 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:29.776317 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8b8448b-c2f2-4e27-bea9-a5d4cd44466f" path="/var/lib/kubelet/pods/a8b8448b-c2f2-4e27-bea9-a5d4cd44466f/volumes" Oct 30 07:19:29.997537 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:29.997508 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:19:29.997967 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:29.997945 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:19:29.998674 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:29.998656 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:19:36.324593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:36.306099 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97"] Oct 30 07:19:36.324593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:36.306334 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" podUID="6cba7319-64a7-453f-95c7-ccf0c18033a0" containerName="istio-proxy" containerID="cri-o://44d32d3e5b94f77719d68955c14e42cef14f989a5db5bd0b064e1f33b23d0846" gracePeriod=30 Oct 30 07:19:38.975269 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:38.975231 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-7fdc99fcb6-bxqx7"] Oct 30 07:19:38.991941 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:38.991913 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-cb54b9dbc-prwtd"] Oct 30 07:19:41.738895 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:41.738808 4951 generic.go:356] "Generic (PLEG): container finished" podID="6cba7319-64a7-453f-95c7-ccf0c18033a0" containerID="44d32d3e5b94f77719d68955c14e42cef14f989a5db5bd0b064e1f33b23d0846" exitCode=0 Oct 30 07:19:41.753605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:41.739630 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" event={"ID":"6cba7319-64a7-453f-95c7-ccf0c18033a0","Type":"ContainerDied","Data":"44d32d3e5b94f77719d68955c14e42cef14f989a5db5bd0b064e1f33b23d0846"} Oct 30 07:19:41.753926 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:41.753898 4951 scope.go:122] "RemoveContainer" containerID="15a6be79554d14c24357a6dc1e551cb55affdfada740b82fee4d6b1982bdb138" Oct 30 07:19:41.914926 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:41.914886 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-438855-f9597c45d-wl76r"] Oct 30 07:19:41.991655 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:41.990328 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-7c7f65f99-ljj28"] Oct 30 07:19:42.092810 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:42.092777 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:19:42.195708 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:42.195674 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-workload-certs\") pod \"6cba7319-64a7-453f-95c7-ccf0c18033a0\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " Oct 30 07:19:42.195907 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:42.195742 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cba7319-64a7-453f-95c7-ccf0c18033a0-istio-token\" (UniqueName: \"kubernetes.io/projected/6cba7319-64a7-453f-95c7-ccf0c18033a0-istio-token\") pod \"6cba7319-64a7-453f-95c7-ccf0c18033a0\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " Oct 30 07:19:42.195907 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:42.195781 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-istio-envoy\") pod \"6cba7319-64a7-453f-95c7-ccf0c18033a0\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " Oct 30 07:19:42.195907 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:42.195848 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-istio-data\") pod \"6cba7319-64a7-453f-95c7-ccf0c18033a0\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " Oct 30 07:19:42.195907 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:42.195896 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cba7319-64a7-453f-95c7-ccf0c18033a0-kube-api-access-7gzfs\" (UniqueName: \"kubernetes.io/projected/6cba7319-64a7-453f-95c7-ccf0c18033a0-kube-api-access-7gzfs\") pod \"6cba7319-64a7-453f-95c7-ccf0c18033a0\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " Oct 30 07:19:42.196139 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:42.195931 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-workload-socket\") pod \"6cba7319-64a7-453f-95c7-ccf0c18033a0\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " Oct 30 07:19:42.196139 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:42.196000 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6cba7319-64a7-453f-95c7-ccf0c18033a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cba7319-64a7-453f-95c7-ccf0c18033a0-istio-podinfo\") pod \"6cba7319-64a7-453f-95c7-ccf0c18033a0\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " Oct 30 07:19:42.196238 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:42.196166 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-credential-socket\") pod \"6cba7319-64a7-453f-95c7-ccf0c18033a0\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " Oct 30 07:19:42.196238 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:42.196220 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cba7319-64a7-453f-95c7-ccf0c18033a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cba7319-64a7-453f-95c7-ccf0c18033a0-istiod-ca-cert\") pod \"6cba7319-64a7-453f-95c7-ccf0c18033a0\" (UID: \"6cba7319-64a7-453f-95c7-ccf0c18033a0\") " Oct 30 07:19:42.196483 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:42.196454 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-istio-data" pod "6cba7319-64a7-453f-95c7-ccf0c18033a0" (UID: "6cba7319-64a7-453f-95c7-ccf0c18033a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:42.196890 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:42.196846 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-workload-certs" pod "6cba7319-64a7-453f-95c7-ccf0c18033a0" (UID: "6cba7319-64a7-453f-95c7-ccf0c18033a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:42.197072 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:42.197049 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cba7319-64a7-453f-95c7-ccf0c18033a0-istiod-ca-cert" pod "6cba7319-64a7-453f-95c7-ccf0c18033a0" (UID: "6cba7319-64a7-453f-95c7-ccf0c18033a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:19:42.197163 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:42.197078 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-workload-socket" pod "6cba7319-64a7-453f-95c7-ccf0c18033a0" (UID: "6cba7319-64a7-453f-95c7-ccf0c18033a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:42.197839 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:42.197432 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cba7319-64a7-453f-95c7-ccf0c18033a0-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:42.198005 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:42.197989 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:42.198132 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:42.198118 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:42.198283 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:42.198268 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:42.198378 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:42.197677 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-credential-socket" pod "6cba7319-64a7-453f-95c7-ccf0c18033a0" (UID: "6cba7319-64a7-453f-95c7-ccf0c18033a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:42.200090 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:42.200053 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cba7319-64a7-453f-95c7-ccf0c18033a0-istio-token" pod "6cba7319-64a7-453f-95c7-ccf0c18033a0" (UID: "6cba7319-64a7-453f-95c7-ccf0c18033a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:42.210242 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:42.201974 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6cba7319-64a7-453f-95c7-ccf0c18033a0-istio-podinfo" pod "6cba7319-64a7-453f-95c7-ccf0c18033a0" (UID: "6cba7319-64a7-453f-95c7-ccf0c18033a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:19:42.210242 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:42.209825 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cba7319-64a7-453f-95c7-ccf0c18033a0-kube-api-access-7gzfs" pod "6cba7319-64a7-453f-95c7-ccf0c18033a0" (UID: "6cba7319-64a7-453f-95c7-ccf0c18033a0"). InnerVolumeSpecName "kube-api-access-7gzfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:42.210603 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:42.210554 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-istio-envoy" pod "6cba7319-64a7-453f-95c7-ccf0c18033a0" (UID: "6cba7319-64a7-453f-95c7-ccf0c18033a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:42.299945 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:42.299912 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cba7319-64a7-453f-95c7-ccf0c18033a0-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:42.299945 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:42.299942 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:42.299945 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:42.299952 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cba7319-64a7-453f-95c7-ccf0c18033a0-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:42.300156 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:42.299960 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cba7319-64a7-453f-95c7-ccf0c18033a0-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:42.300156 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:42.299969 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gzfs\" (UniqueName: \"kubernetes.io/projected/6cba7319-64a7-453f-95c7-ccf0c18033a0-kube-api-access-7gzfs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:42.818765 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:42.818726 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" event={"ID":"6cba7319-64a7-453f-95c7-ccf0c18033a0","Type":"ContainerDied","Data":"3ab768ce7306223007af6009c8630d7c338e4e29e4d987c0200f6dcb8fb10aca"} Oct 30 07:19:42.819613 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:42.819590 4951 scope.go:122] "RemoveContainer" containerID="44d32d3e5b94f77719d68955c14e42cef14f989a5db5bd0b064e1f33b23d0846" Oct 30 07:19:42.838756 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:42.819510 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97" Oct 30 07:19:42.915531 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:42.915487 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97"] Oct 30 07:19:42.927970 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:42.927940 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-6cc5f2-openshift-default-7cfb6f55f5-vzt97"] Oct 30 07:19:43.777208 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:43.777173 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cba7319-64a7-453f-95c7-ccf0c18033a0" path="/var/lib/kubelet/pods/6cba7319-64a7-453f-95c7-ccf0c18033a0/volumes" Oct 30 07:19:43.975832 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:43.975781 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-7fdc99fcb6-bxqx7" podUID="ed6e5a68-478f-4bad-8a26-1a9aecea9672" containerName="payload-processing" containerID="cri-o://5c55073ad4c961162068b47bb7461a107d76af92d7e455988ec548ddef2bcde7" gracePeriod=25 Oct 30 07:19:43.992864 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:43.992779 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-cb54b9dbc-prwtd" podUID="730c927a-96d9-41ef-9e20-b44af912068e" containerName="payload-pre-processing" containerID="cri-o://74be6be526959c4f2073cf8b40002e347a9c4f53ee41b3f18f7c364fda794027" gracePeriod=25 Oct 30 07:19:44.361418 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:44.361120 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-7fdc99fcb6-bxqx7" Oct 30 07:19:44.467505 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:44.467464 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-cb54b9dbc-prwtd" Oct 30 07:19:44.559522 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:44.559324 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed6e5a68-478f-4bad-8a26-1a9aecea9672-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed6e5a68-478f-4bad-8a26-1a9aecea9672-plugins-config-volume\") pod \"ed6e5a68-478f-4bad-8a26-1a9aecea9672\" (UID: \"ed6e5a68-478f-4bad-8a26-1a9aecea9672\") " Oct 30 07:19:44.559801 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:44.559744 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed6e5a68-478f-4bad-8a26-1a9aecea9672-plugins-config-volume" pod "ed6e5a68-478f-4bad-8a26-1a9aecea9672" (UID: "ed6e5a68-478f-4bad-8a26-1a9aecea9672"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:19:44.560145 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:44.560109 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed6e5a68-478f-4bad-8a26-1a9aecea9672-kube-api-access-vbjwc\" (UniqueName: \"kubernetes.io/projected/ed6e5a68-478f-4bad-8a26-1a9aecea9672-kube-api-access-vbjwc\") pod \"ed6e5a68-478f-4bad-8a26-1a9aecea9672\" (UID: \"ed6e5a68-478f-4bad-8a26-1a9aecea9672\") " Oct 30 07:19:44.560455 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:44.560415 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/730c927a-96d9-41ef-9e20-b44af912068e-kube-api-access-6qqcx\" (UniqueName: \"kubernetes.io/projected/730c927a-96d9-41ef-9e20-b44af912068e-kube-api-access-6qqcx\") pod \"730c927a-96d9-41ef-9e20-b44af912068e\" (UID: \"730c927a-96d9-41ef-9e20-b44af912068e\") " Oct 30 07:19:44.560695 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:44.560659 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/730c927a-96d9-41ef-9e20-b44af912068e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730c927a-96d9-41ef-9e20-b44af912068e-plugins-config-volume\") pod \"730c927a-96d9-41ef-9e20-b44af912068e\" (UID: \"730c927a-96d9-41ef-9e20-b44af912068e\") " Oct 30 07:19:44.561616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:44.561551 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed6e5a68-478f-4bad-8a26-1a9aecea9672-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:44.562121 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:44.562025 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/730c927a-96d9-41ef-9e20-b44af912068e-plugins-config-volume" pod "730c927a-96d9-41ef-9e20-b44af912068e" (UID: "730c927a-96d9-41ef-9e20-b44af912068e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:19:44.564293 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:44.564257 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/730c927a-96d9-41ef-9e20-b44af912068e-kube-api-access-6qqcx" pod "730c927a-96d9-41ef-9e20-b44af912068e" (UID: "730c927a-96d9-41ef-9e20-b44af912068e"). InnerVolumeSpecName "kube-api-access-6qqcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:44.573940 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:44.573912 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed6e5a68-478f-4bad-8a26-1a9aecea9672-kube-api-access-vbjwc" pod "ed6e5a68-478f-4bad-8a26-1a9aecea9672" (UID: "ed6e5a68-478f-4bad-8a26-1a9aecea9672"). InnerVolumeSpecName "kube-api-access-vbjwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:44.663473 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:44.663383 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qqcx\" (UniqueName: \"kubernetes.io/projected/730c927a-96d9-41ef-9e20-b44af912068e-kube-api-access-6qqcx\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:44.663473 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:44.663424 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730c927a-96d9-41ef-9e20-b44af912068e-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:44.663473 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:44.663439 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbjwc\" (UniqueName: \"kubernetes.io/projected/ed6e5a68-478f-4bad-8a26-1a9aecea9672-kube-api-access-vbjwc\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:44.869627 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:44.869585 4951 generic.go:356] "Generic (PLEG): container finished" podID="730c927a-96d9-41ef-9e20-b44af912068e" containerID="74be6be526959c4f2073cf8b40002e347a9c4f53ee41b3f18f7c364fda794027" exitCode=0 Oct 30 07:19:44.869826 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:44.869732 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-cb54b9dbc-prwtd" event={"ID":"730c927a-96d9-41ef-9e20-b44af912068e","Type":"ContainerDied","Data":"74be6be526959c4f2073cf8b40002e347a9c4f53ee41b3f18f7c364fda794027"} Oct 30 07:19:44.869826 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:44.869789 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-cb54b9dbc-prwtd" event={"ID":"730c927a-96d9-41ef-9e20-b44af912068e","Type":"ContainerDied","Data":"e26fa592090f0a0c56b6a14b870ba4ef08f33b4a40b252fd7b15a3db1f9923be"} Oct 30 07:19:44.869826 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:44.869819 4951 scope.go:122] "RemoveContainer" containerID="74be6be526959c4f2073cf8b40002e347a9c4f53ee41b3f18f7c364fda794027" Oct 30 07:19:44.869999 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:44.869977 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-cb54b9dbc-prwtd" Oct 30 07:19:44.876412 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:44.876368 4951 generic.go:356] "Generic (PLEG): container finished" podID="ed6e5a68-478f-4bad-8a26-1a9aecea9672" containerID="5c55073ad4c961162068b47bb7461a107d76af92d7e455988ec548ddef2bcde7" exitCode=0 Oct 30 07:19:44.876732 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:44.876710 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-7fdc99fcb6-bxqx7" event={"ID":"ed6e5a68-478f-4bad-8a26-1a9aecea9672","Type":"ContainerDied","Data":"5c55073ad4c961162068b47bb7461a107d76af92d7e455988ec548ddef2bcde7"} Oct 30 07:19:44.877013 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:44.876902 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-7fdc99fcb6-bxqx7" event={"ID":"ed6e5a68-478f-4bad-8a26-1a9aecea9672","Type":"ContainerDied","Data":"561f1b3ebcdb405b6bc81941d7923ef06e23c2b244f21f617fbbbeb14c488730"} Oct 30 07:19:44.877201 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:44.877136 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-7fdc99fcb6-bxqx7" Oct 30 07:19:45.062364 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:45.062322 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-7fdc99fcb6-bxqx7"] Oct 30 07:19:45.117643 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:45.117244 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-7fdc99fcb6-bxqx7"] Oct 30 07:19:45.134302 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:45.134262 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-cb54b9dbc-prwtd"] Oct 30 07:19:45.139699 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:45.139665 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-cb54b9dbc-prwtd"] Oct 30 07:19:45.151132 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:45.150976 4951 scope.go:122] "RemoveContainer" containerID="74be6be526959c4f2073cf8b40002e347a9c4f53ee41b3f18f7c364fda794027" Oct 30 07:19:45.152784 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:19:45.152602 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74be6be526959c4f2073cf8b40002e347a9c4f53ee41b3f18f7c364fda794027\": container with ID starting with 74be6be526959c4f2073cf8b40002e347a9c4f53ee41b3f18f7c364fda794027 not found: ID does not exist" containerID="74be6be526959c4f2073cf8b40002e347a9c4f53ee41b3f18f7c364fda794027" Oct 30 07:19:45.152784 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:45.152646 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74be6be526959c4f2073cf8b40002e347a9c4f53ee41b3f18f7c364fda794027"} err="failed to get container status \"74be6be526959c4f2073cf8b40002e347a9c4f53ee41b3f18f7c364fda794027\": rpc error: code = NotFound desc = could not find container \"74be6be526959c4f2073cf8b40002e347a9c4f53ee41b3f18f7c364fda794027\": container with ID starting with 74be6be526959c4f2073cf8b40002e347a9c4f53ee41b3f18f7c364fda794027 not found: ID does not exist" Oct 30 07:19:45.152784 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:45.152672 4951 scope.go:122] "RemoveContainer" containerID="5c55073ad4c961162068b47bb7461a107d76af92d7e455988ec548ddef2bcde7" Oct 30 07:19:45.173101 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:45.173072 4951 scope.go:122] "RemoveContainer" containerID="5c55073ad4c961162068b47bb7461a107d76af92d7e455988ec548ddef2bcde7" Oct 30 07:19:45.173832 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:19:45.173745 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c55073ad4c961162068b47bb7461a107d76af92d7e455988ec548ddef2bcde7\": container with ID starting with 5c55073ad4c961162068b47bb7461a107d76af92d7e455988ec548ddef2bcde7 not found: ID does not exist" containerID="5c55073ad4c961162068b47bb7461a107d76af92d7e455988ec548ddef2bcde7" Oct 30 07:19:45.173832 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:45.173785 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c55073ad4c961162068b47bb7461a107d76af92d7e455988ec548ddef2bcde7"} err="failed to get container status \"5c55073ad4c961162068b47bb7461a107d76af92d7e455988ec548ddef2bcde7\": rpc error: code = NotFound desc = could not find container \"5c55073ad4c961162068b47bb7461a107d76af92d7e455988ec548ddef2bcde7\": container with ID starting with 5c55073ad4c961162068b47bb7461a107d76af92d7e455988ec548ddef2bcde7 not found: ID does not exist" Oct 30 07:19:45.722933 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:45.722824 4951 patch_prober.go:29] interesting pod/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:19:45.722933 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:45.722911 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" podUID="011a2449-9f58-427a-aa6e-89e4a911c2e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:19:45.777046 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:45.777005 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="730c927a-96d9-41ef-9e20-b44af912068e" path="/var/lib/kubelet/pods/730c927a-96d9-41ef-9e20-b44af912068e/volumes" Oct 30 07:19:45.777691 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:45.777669 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed6e5a68-478f-4bad-8a26-1a9aecea9672" path="/var/lib/kubelet/pods/ed6e5a68-478f-4bad-8a26-1a9aecea9672/volumes" Oct 30 07:19:46.915705 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:46.915664 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-f9597c45d-wl76r" podUID="126b094f-2a13-4eff-bd0b-3c0e2a52853e" containerName="payload-processing" containerID="cri-o://613de3349c9c44ab5950b257b9e773099875cf7c35ac9f3963d2e864f179ddb2" gracePeriod=25 Oct 30 07:19:46.991286 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:46.991244 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-7c7f65f99-ljj28" podUID="7e0f9017-d26f-4aff-99ef-c07825f692ad" containerName="payload-pre-processing" containerID="cri-o://8c079531fdff293e0dca53e80a982c0d2acd45f76c958129254bcb0b6b95d51b" gracePeriod=25 Oct 30 07:19:47.458446 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.458379 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-7c7f65f99-ljj28" Oct 30 07:19:47.463416 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.463390 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-f9597c45d-wl76r" Oct 30 07:19:47.493999 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.493956 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-486cd4-5db6fd58-tg4wr"] Oct 30 07:19:47.533919 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.533874 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-486cd4-5758d6dfc9-v5nfj"] Oct 30 07:19:47.605372 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.603392 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/126b094f-2a13-4eff-bd0b-3c0e2a52853e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/126b094f-2a13-4eff-bd0b-3c0e2a52853e-plugins-config-volume\") pod \"126b094f-2a13-4eff-bd0b-3c0e2a52853e\" (UID: \"126b094f-2a13-4eff-bd0b-3c0e2a52853e\") " Oct 30 07:19:47.605372 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.603515 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/126b094f-2a13-4eff-bd0b-3c0e2a52853e-kube-api-access-6cdqb\" (UniqueName: \"kubernetes.io/projected/126b094f-2a13-4eff-bd0b-3c0e2a52853e-kube-api-access-6cdqb\") pod \"126b094f-2a13-4eff-bd0b-3c0e2a52853e\" (UID: \"126b094f-2a13-4eff-bd0b-3c0e2a52853e\") " Oct 30 07:19:47.605372 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.603581 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e0f9017-d26f-4aff-99ef-c07825f692ad-kube-api-access-rm4nq\" (UniqueName: \"kubernetes.io/projected/7e0f9017-d26f-4aff-99ef-c07825f692ad-kube-api-access-rm4nq\") pod \"7e0f9017-d26f-4aff-99ef-c07825f692ad\" (UID: \"7e0f9017-d26f-4aff-99ef-c07825f692ad\") " Oct 30 07:19:47.605372 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.603629 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e0f9017-d26f-4aff-99ef-c07825f692ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e0f9017-d26f-4aff-99ef-c07825f692ad-plugins-config-volume\") pod \"7e0f9017-d26f-4aff-99ef-c07825f692ad\" (UID: \"7e0f9017-d26f-4aff-99ef-c07825f692ad\") " Oct 30 07:19:47.605372 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.604696 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e0f9017-d26f-4aff-99ef-c07825f692ad-plugins-config-volume" pod "7e0f9017-d26f-4aff-99ef-c07825f692ad" (UID: "7e0f9017-d26f-4aff-99ef-c07825f692ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:19:47.606616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.606547 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/126b094f-2a13-4eff-bd0b-3c0e2a52853e-plugins-config-volume" pod "126b094f-2a13-4eff-bd0b-3c0e2a52853e" (UID: "126b094f-2a13-4eff-bd0b-3c0e2a52853e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:19:47.608881 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.608857 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0f9017-d26f-4aff-99ef-c07825f692ad-kube-api-access-rm4nq" pod "7e0f9017-d26f-4aff-99ef-c07825f692ad" (UID: "7e0f9017-d26f-4aff-99ef-c07825f692ad"). InnerVolumeSpecName "kube-api-access-rm4nq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:47.609997 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.609974 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/126b094f-2a13-4eff-bd0b-3c0e2a52853e-kube-api-access-6cdqb" pod "126b094f-2a13-4eff-bd0b-3c0e2a52853e" (UID: "126b094f-2a13-4eff-bd0b-3c0e2a52853e"). InnerVolumeSpecName "kube-api-access-6cdqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:47.705749 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.705717 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cdqb\" (UniqueName: \"kubernetes.io/projected/126b094f-2a13-4eff-bd0b-3c0e2a52853e-kube-api-access-6cdqb\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:47.705749 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.705746 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm4nq\" (UniqueName: \"kubernetes.io/projected/7e0f9017-d26f-4aff-99ef-c07825f692ad-kube-api-access-rm4nq\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:47.705749 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.705756 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e0f9017-d26f-4aff-99ef-c07825f692ad-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:47.705988 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.705765 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/126b094f-2a13-4eff-bd0b-3c0e2a52853e-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:47.905019 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.904989 4951 generic.go:356] "Generic (PLEG): container finished" podID="2898e6e0-5229-4c67-b3e3-8f2076284b86" containerID="1e43449f9e6c7034fa598e65aef6052612b73a058c0388f7e0fd1239aa6afb21" exitCode=0 Oct 30 07:19:47.905195 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.905064 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" event={"ID":"2898e6e0-5229-4c67-b3e3-8f2076284b86","Type":"ContainerDied","Data":"1e43449f9e6c7034fa598e65aef6052612b73a058c0388f7e0fd1239aa6afb21"} Oct 30 07:19:47.905626 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.905606 4951 scope.go:122] "RemoveContainer" containerID="1e43449f9e6c7034fa598e65aef6052612b73a058c0388f7e0fd1239aa6afb21" Oct 30 07:19:47.907282 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.907249 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:19:47.907393 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.907322 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:19:47.907393 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.907385 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:19:47.907483 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.907460 4951 generic.go:356] "Generic (PLEG): container finished" podID="011a2449-9f58-427a-aa6e-89e4a911c2e5" containerID="b8f3461c9a0c2a6751c03305a15a997e9dc65e63ebf891040e7c6c33e984cf2a" exitCode=0 Oct 30 07:19:47.907555 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.907536 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" event={"ID":"011a2449-9f58-427a-aa6e-89e4a911c2e5","Type":"ContainerDied","Data":"b8f3461c9a0c2a6751c03305a15a997e9dc65e63ebf891040e7c6c33e984cf2a"} Oct 30 07:19:47.908035 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.907948 4951 scope.go:122] "RemoveContainer" containerID="b8f3461c9a0c2a6751c03305a15a997e9dc65e63ebf891040e7c6c33e984cf2a" Oct 30 07:19:47.909466 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.909439 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:19:47.909535 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.909501 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:19:47.909609 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.909534 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:19:47.910812 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.910793 4951 generic.go:356] "Generic (PLEG): container finished" podID="126b094f-2a13-4eff-bd0b-3c0e2a52853e" containerID="613de3349c9c44ab5950b257b9e773099875cf7c35ac9f3963d2e864f179ddb2" exitCode=0 Oct 30 07:19:47.910898 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.910884 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-f9597c45d-wl76r" event={"ID":"126b094f-2a13-4eff-bd0b-3c0e2a52853e","Type":"ContainerDied","Data":"613de3349c9c44ab5950b257b9e773099875cf7c35ac9f3963d2e864f179ddb2"} Oct 30 07:19:47.910944 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.910888 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-f9597c45d-wl76r" Oct 30 07:19:47.910944 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.910926 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-f9597c45d-wl76r" event={"ID":"126b094f-2a13-4eff-bd0b-3c0e2a52853e","Type":"ContainerDied","Data":"c76c4d7021d8964ca227f0ab3628c3fa4ae30f52c4dc67149b5c2cb28a7ce731"} Oct 30 07:19:47.911039 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.910947 4951 scope.go:122] "RemoveContainer" containerID="613de3349c9c44ab5950b257b9e773099875cf7c35ac9f3963d2e864f179ddb2" Oct 30 07:19:47.918754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.918566 4951 generic.go:356] "Generic (PLEG): container finished" podID="7e0f9017-d26f-4aff-99ef-c07825f692ad" containerID="8c079531fdff293e0dca53e80a982c0d2acd45f76c958129254bcb0b6b95d51b" exitCode=0 Oct 30 07:19:47.918754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.918691 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-7c7f65f99-ljj28" event={"ID":"7e0f9017-d26f-4aff-99ef-c07825f692ad","Type":"ContainerDied","Data":"8c079531fdff293e0dca53e80a982c0d2acd45f76c958129254bcb0b6b95d51b"} Oct 30 07:19:47.918754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.918730 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-7c7f65f99-ljj28" Oct 30 07:19:47.919530 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.918734 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-7c7f65f99-ljj28" event={"ID":"7e0f9017-d26f-4aff-99ef-c07825f692ad","Type":"ContainerDied","Data":"445a54bc40a324ebc410088a6e5c1e4b5fca2d0f49fa6eeef73e01f3842938bd"} Oct 30 07:19:47.941381 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.941355 4951 scope.go:122] "RemoveContainer" containerID="613de3349c9c44ab5950b257b9e773099875cf7c35ac9f3963d2e864f179ddb2" Oct 30 07:19:47.942103 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:19:47.942077 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"613de3349c9c44ab5950b257b9e773099875cf7c35ac9f3963d2e864f179ddb2\": container with ID starting with 613de3349c9c44ab5950b257b9e773099875cf7c35ac9f3963d2e864f179ddb2 not found: ID does not exist" containerID="613de3349c9c44ab5950b257b9e773099875cf7c35ac9f3963d2e864f179ddb2" Oct 30 07:19:47.942212 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.942113 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613de3349c9c44ab5950b257b9e773099875cf7c35ac9f3963d2e864f179ddb2"} err="failed to get container status \"613de3349c9c44ab5950b257b9e773099875cf7c35ac9f3963d2e864f179ddb2\": rpc error: code = NotFound desc = could not find container \"613de3349c9c44ab5950b257b9e773099875cf7c35ac9f3963d2e864f179ddb2\": container with ID starting with 613de3349c9c44ab5950b257b9e773099875cf7c35ac9f3963d2e864f179ddb2 not found: ID does not exist" Oct 30 07:19:47.942212 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.942136 4951 scope.go:122] "RemoveContainer" containerID="8c079531fdff293e0dca53e80a982c0d2acd45f76c958129254bcb0b6b95d51b" Oct 30 07:19:47.967473 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.966675 4951 scope.go:122] "RemoveContainer" containerID="8c079531fdff293e0dca53e80a982c0d2acd45f76c958129254bcb0b6b95d51b" Oct 30 07:19:47.967945 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:19:47.967877 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c079531fdff293e0dca53e80a982c0d2acd45f76c958129254bcb0b6b95d51b\": container with ID starting with 8c079531fdff293e0dca53e80a982c0d2acd45f76c958129254bcb0b6b95d51b not found: ID does not exist" containerID="8c079531fdff293e0dca53e80a982c0d2acd45f76c958129254bcb0b6b95d51b" Oct 30 07:19:47.967945 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:47.967912 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c079531fdff293e0dca53e80a982c0d2acd45f76c958129254bcb0b6b95d51b"} err="failed to get container status \"8c079531fdff293e0dca53e80a982c0d2acd45f76c958129254bcb0b6b95d51b\": rpc error: code = NotFound desc = could not find container \"8c079531fdff293e0dca53e80a982c0d2acd45f76c958129254bcb0b6b95d51b\": container with ID starting with 8c079531fdff293e0dca53e80a982c0d2acd45f76c958129254bcb0b6b95d51b not found: ID does not exist" Oct 30 07:19:48.067829 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:48.067793 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-438855-f9597c45d-wl76r"] Oct 30 07:19:48.072705 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:48.072673 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-438855-f9597c45d-wl76r"] Oct 30 07:19:48.101689 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:48.096721 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-7c7f65f99-ljj28"] Oct 30 07:19:48.101689 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:48.100529 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-7c7f65f99-ljj28"] Oct 30 07:19:48.946333 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:48.946288 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" event={"ID":"011a2449-9f58-427a-aa6e-89e4a911c2e5","Type":"ContainerStarted","Data":"7da231a53c77ced45e7f3c7b4ba8fb454a91b7e3a3689b0b4a1f986513ef9c51"} Oct 30 07:19:48.954630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:48.954601 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" event={"ID":"2898e6e0-5229-4c67-b3e3-8f2076284b86","Type":"ContainerStarted","Data":"b570b4b4afd8fa284d9ebf9664956f3376d292c4eae79b456106c29d36dd0348"} Oct 30 07:19:49.546597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:49.545952 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:19:49.546597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:49.546111 4951 patch_prober.go:29] interesting pod/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 07:19:49.546597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:49.546153 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" podUID="011a2449-9f58-427a-aa6e-89e4a911c2e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 07:19:49.607101 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:49.607001 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:19:49.607357 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:49.607337 4951 patch_prober.go:29] interesting pod/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 07:19:49.607488 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:49.607470 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" podUID="2898e6e0-5229-4c67-b3e3-8f2076284b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 07:19:49.795737 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:49.795705 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="126b094f-2a13-4eff-bd0b-3c0e2a52853e" path="/var/lib/kubelet/pods/126b094f-2a13-4eff-bd0b-3c0e2a52853e/volumes" Oct 30 07:19:49.796135 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:49.796120 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e0f9017-d26f-4aff-99ef-c07825f692ad" path="/var/lib/kubelet/pods/7e0f9017-d26f-4aff-99ef-c07825f692ad/volumes" Oct 30 07:19:50.544718 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:50.544681 4951 patch_prober.go:29] interesting pod/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 07:19:50.545345 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:50.545311 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" podUID="011a2449-9f58-427a-aa6e-89e4a911c2e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 07:19:50.614292 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:50.614237 4951 patch_prober.go:29] interesting pod/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 07:19:50.614771 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:50.614637 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" podUID="2898e6e0-5229-4c67-b3e3-8f2076284b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 07:19:51.560044 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:51.560004 4951 patch_prober.go:29] interesting pod/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 07:19:51.560672 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:51.560635 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" podUID="011a2449-9f58-427a-aa6e-89e4a911c2e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 07:19:51.607287 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:51.607256 4951 patch_prober.go:29] interesting pod/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 07:19:51.607490 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:51.607470 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" podUID="2898e6e0-5229-4c67-b3e3-8f2076284b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 07:19:52.504746 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:52.504657 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-486cd4-5db6fd58-tg4wr" podUID="10fee9bf-b760-42ee-bd0c-93a44cee033e" containerName="payload-processing" containerID="cri-o://1714b0aebfb2a59dbead75147b9edf8e81362625c97b77313291e98c1a6089a2" gracePeriod=25 Oct 30 07:19:52.537333 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:52.537281 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-486cd4-5758d6dfc9-v5nfj" podUID="8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1" containerName="payload-pre-processing" containerID="cri-o://c0541eafaad5dae72820871097d23c1509ce657ef20fc4ee509f6dc2910abd58" gracePeriod=25 Oct 30 07:19:52.557596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:52.545970 4951 patch_prober.go:29] interesting pod/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 07:19:52.557596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:52.546032 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" podUID="011a2449-9f58-427a-aa6e-89e4a911c2e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 07:19:52.612622 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:52.607440 4951 patch_prober.go:29] interesting pod/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 07:19:52.612622 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:52.607502 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" podUID="2898e6e0-5229-4c67-b3e3-8f2076284b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 07:19:53.083594 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:19:53.081452 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:19:53.104274 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.101968 4951 generic.go:356] "Generic (PLEG): container finished" podID="8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1" containerID="c0541eafaad5dae72820871097d23c1509ce657ef20fc4ee509f6dc2910abd58" exitCode=0 Oct 30 07:19:53.104274 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.102080 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-486cd4-5758d6dfc9-v5nfj" event={"ID":"8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1","Type":"ContainerDied","Data":"c0541eafaad5dae72820871097d23c1509ce657ef20fc4ee509f6dc2910abd58"} Oct 30 07:19:53.106321 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.105160 4951 generic.go:356] "Generic (PLEG): container finished" podID="10fee9bf-b760-42ee-bd0c-93a44cee033e" containerID="1714b0aebfb2a59dbead75147b9edf8e81362625c97b77313291e98c1a6089a2" exitCode=0 Oct 30 07:19:53.106321 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.105331 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-486cd4-5db6fd58-tg4wr" event={"ID":"10fee9bf-b760-42ee-bd0c-93a44cee033e","Type":"ContainerDied","Data":"1714b0aebfb2a59dbead75147b9edf8e81362625c97b77313291e98c1a6089a2"} Oct 30 07:19:53.106321 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.105381 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-486cd4-5db6fd58-tg4wr" event={"ID":"10fee9bf-b760-42ee-bd0c-93a44cee033e","Type":"ContainerDied","Data":"0bfb00d72f1efb00d7846ebf2dc6b11d26f347c91e421765a82f02ed4691349f"} Oct 30 07:19:53.106321 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.105405 4951 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bfb00d72f1efb00d7846ebf2dc6b11d26f347c91e421765a82f02ed4691349f" Oct 30 07:19:53.113782 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.113744 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-486cd4-5db6fd58-tg4wr" Oct 30 07:19:53.209644 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.207029 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10fee9bf-b760-42ee-bd0c-93a44cee033e-kube-api-access-8n9mr\" (UniqueName: \"kubernetes.io/projected/10fee9bf-b760-42ee-bd0c-93a44cee033e-kube-api-access-8n9mr\") pod \"10fee9bf-b760-42ee-bd0c-93a44cee033e\" (UID: \"10fee9bf-b760-42ee-bd0c-93a44cee033e\") " Oct 30 07:19:53.209644 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.207275 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10fee9bf-b760-42ee-bd0c-93a44cee033e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10fee9bf-b760-42ee-bd0c-93a44cee033e-plugins-config-volume\") pod \"10fee9bf-b760-42ee-bd0c-93a44cee033e\" (UID: \"10fee9bf-b760-42ee-bd0c-93a44cee033e\") " Oct 30 07:19:53.209644 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.208664 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10fee9bf-b760-42ee-bd0c-93a44cee033e-plugins-config-volume" pod "10fee9bf-b760-42ee-bd0c-93a44cee033e" (UID: "10fee9bf-b760-42ee-bd0c-93a44cee033e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:19:53.210972 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.210943 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10fee9bf-b760-42ee-bd0c-93a44cee033e-kube-api-access-8n9mr" pod "10fee9bf-b760-42ee-bd0c-93a44cee033e" (UID: "10fee9bf-b760-42ee-bd0c-93a44cee033e"). InnerVolumeSpecName "kube-api-access-8n9mr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:53.309464 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.309328 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-486cd4-5758d6dfc9-v5nfj" Oct 30 07:19:53.310790 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.310767 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n9mr\" (UniqueName: \"kubernetes.io/projected/10fee9bf-b760-42ee-bd0c-93a44cee033e-kube-api-access-8n9mr\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:53.310875 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.310795 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10fee9bf-b760-42ee-bd0c-93a44cee033e-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:53.411566 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.411527 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1-plugins-config-volume\") pod \"8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1\" (UID: \"8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1\") " Oct 30 07:19:53.411884 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.411659 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1-kube-api-access-gwhkg\" (UniqueName: \"kubernetes.io/projected/8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1-kube-api-access-gwhkg\") pod \"8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1\" (UID: \"8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1\") " Oct 30 07:19:53.412025 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.412000 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1-plugins-config-volume" pod "8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1" (UID: "8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:19:53.412364 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.412315 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:53.414693 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.414658 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1-kube-api-access-gwhkg" pod "8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1" (UID: "8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1"). InnerVolumeSpecName "kube-api-access-gwhkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:53.513863 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.513812 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwhkg\" (UniqueName: \"kubernetes.io/projected/8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1-kube-api-access-gwhkg\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:19:53.543436 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.542972 4951 patch_prober.go:29] interesting pod/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 07:19:53.543436 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.543040 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" podUID="011a2449-9f58-427a-aa6e-89e4a911c2e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 07:19:53.606733 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.606698 4951 patch_prober.go:29] interesting pod/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 07:19:53.606861 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.606758 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" podUID="2898e6e0-5229-4c67-b3e3-8f2076284b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 07:19:53.996647 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.995888 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx"] Oct 30 07:19:53.997126 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.996898 4951 cpu_manager.go:414] "containerMap: removing container" podUID="a8b8448b-c2f2-4e27-bea9-a5d4cd44466f" containerName="maas-api" Oct 30 07:19:53.997126 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.996921 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b8448b-c2f2-4e27-bea9-a5d4cd44466f" containerName="maas-api" Oct 30 07:19:53.997126 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.996936 4951 cpu_manager.go:414] "containerMap: removing container" podUID="8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1" containerName="payload-pre-processing" Oct 30 07:19:53.997126 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.996943 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1" containerName="payload-pre-processing" Oct 30 07:19:53.997126 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.996963 4951 cpu_manager.go:414] "containerMap: removing container" podUID="d159ec3e-d629-4a66-b0c7-17d00455c088" containerName="istio-proxy" Oct 30 07:19:53.997126 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.996972 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d159ec3e-d629-4a66-b0c7-17d00455c088" containerName="istio-proxy" Oct 30 07:19:53.997126 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.996988 4951 cpu_manager.go:414] "containerMap: removing container" podUID="7e0f9017-d26f-4aff-99ef-c07825f692ad" containerName="payload-pre-processing" Oct 30 07:19:53.997126 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.996996 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e0f9017-d26f-4aff-99ef-c07825f692ad" containerName="payload-pre-processing" Oct 30 07:19:53.997126 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.997012 4951 cpu_manager.go:414] "containerMap: removing container" podUID="6cba7319-64a7-453f-95c7-ccf0c18033a0" containerName="istio-proxy" Oct 30 07:19:53.997126 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.997020 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cba7319-64a7-453f-95c7-ccf0c18033a0" containerName="istio-proxy" Oct 30 07:19:53.997126 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.997035 4951 cpu_manager.go:414] "containerMap: removing container" podUID="5c9605f5-3aa1-4f48-8ec6-1b709863952f" containerName="istio-proxy" Oct 30 07:19:53.997126 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.997042 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c9605f5-3aa1-4f48-8ec6-1b709863952f" containerName="istio-proxy" Oct 30 07:19:53.997126 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.997059 4951 cpu_manager.go:414] "containerMap: removing container" podUID="6cba7319-64a7-453f-95c7-ccf0c18033a0" containerName="istio-proxy" Oct 30 07:19:53.997126 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.997067 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cba7319-64a7-453f-95c7-ccf0c18033a0" containerName="istio-proxy" Oct 30 07:19:53.997126 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.997079 4951 cpu_manager.go:414] "containerMap: removing container" podUID="ed6e5a68-478f-4bad-8a26-1a9aecea9672" containerName="payload-processing" Oct 30 07:19:53.997126 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.997087 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed6e5a68-478f-4bad-8a26-1a9aecea9672" containerName="payload-processing" Oct 30 07:19:53.997126 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.997099 4951 cpu_manager.go:414] "containerMap: removing container" podUID="10fee9bf-b760-42ee-bd0c-93a44cee033e" containerName="payload-processing" Oct 30 07:19:53.997126 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.997106 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10fee9bf-b760-42ee-bd0c-93a44cee033e" containerName="payload-processing" Oct 30 07:19:53.997126 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.997129 4951 cpu_manager.go:414] "containerMap: removing container" podUID="89d560f3-70af-44f7-8084-4ca608698965" containerName="istio-proxy" Oct 30 07:19:53.997126 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.997136 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89d560f3-70af-44f7-8084-4ca608698965" containerName="istio-proxy" Oct 30 07:19:53.998124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.997149 4951 cpu_manager.go:414] "containerMap: removing container" podUID="98e03afc-4edf-499e-a63a-0f92372c65a9" containerName="istio-proxy" Oct 30 07:19:53.998124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.997156 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98e03afc-4edf-499e-a63a-0f92372c65a9" containerName="istio-proxy" Oct 30 07:19:53.998124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.997195 4951 cpu_manager.go:414] "containerMap: removing container" podUID="126b094f-2a13-4eff-bd0b-3c0e2a52853e" containerName="payload-processing" Oct 30 07:19:53.998124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.997202 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="126b094f-2a13-4eff-bd0b-3c0e2a52853e" containerName="payload-processing" Oct 30 07:19:53.998124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.997218 4951 cpu_manager.go:414] "containerMap: removing container" podUID="730c927a-96d9-41ef-9e20-b44af912068e" containerName="payload-pre-processing" Oct 30 07:19:53.998124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.997225 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="730c927a-96d9-41ef-9e20-b44af912068e" containerName="payload-pre-processing" Oct 30 07:19:53.998124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.997376 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="10fee9bf-b760-42ee-bd0c-93a44cee033e" containerName="payload-processing" Oct 30 07:19:53.998124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.997391 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="89d560f3-70af-44f7-8084-4ca608698965" containerName="istio-proxy" Oct 30 07:19:53.998124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.997402 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="98e03afc-4edf-499e-a63a-0f92372c65a9" containerName="istio-proxy" Oct 30 07:19:53.998124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.997418 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="126b094f-2a13-4eff-bd0b-3c0e2a52853e" containerName="payload-processing" Oct 30 07:19:53.998124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.997427 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="730c927a-96d9-41ef-9e20-b44af912068e" containerName="payload-pre-processing" Oct 30 07:19:53.998124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.997440 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b8448b-c2f2-4e27-bea9-a5d4cd44466f" containerName="maas-api" Oct 30 07:19:53.998124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.997448 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1" containerName="payload-pre-processing" Oct 30 07:19:53.998124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.997459 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="d159ec3e-d629-4a66-b0c7-17d00455c088" containerName="istio-proxy" Oct 30 07:19:53.998124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.997468 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e0f9017-d26f-4aff-99ef-c07825f692ad" containerName="payload-pre-processing" Oct 30 07:19:53.998124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.997479 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cba7319-64a7-453f-95c7-ccf0c18033a0" containerName="istio-proxy" Oct 30 07:19:53.998124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.997487 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c9605f5-3aa1-4f48-8ec6-1b709863952f" containerName="istio-proxy" Oct 30 07:19:53.998124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.997499 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cba7319-64a7-453f-95c7-ccf0c18033a0" containerName="istio-proxy" Oct 30 07:19:53.998124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:53.997508 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed6e5a68-478f-4bad-8a26-1a9aecea9672" containerName="payload-processing" Oct 30 07:19:54.004594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.003403 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:19:54.030808 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.022139 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istio-data\") pod \"e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:19:54.030808 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.022197 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istio-token\") pod \"e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:19:54.030808 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.022262 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istiod-ca-cert\") pod \"e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:19:54.030808 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.022399 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-credential-socket\") pod \"e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:19:54.030808 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.022456 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-workload-certs\") pod \"e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:19:54.030808 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.022478 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istio-envoy\") pod \"e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:19:54.030808 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.022538 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-workload-socket\") pod \"e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:19:54.030808 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.022558 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istio-podinfo\") pod \"e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:19:54.036591 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.031772 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zfhw\" (UniqueName: \"kubernetes.io/projected/0c51720b-1e0b-4cb0-a1e2-08065c91acde-kube-api-access-8zfhw\") pod \"e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:19:54.068596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.045802 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-ab3d83-openshift-default-dockercfg-ghp2t\"" Oct 30 07:19:54.092709 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.092673 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx"] Oct 30 07:19:54.112660 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.112633 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-486cd4-5758d6dfc9-v5nfj" Oct 30 07:19:54.112860 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.112825 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-486cd4-5db6fd58-tg4wr" Oct 30 07:19:54.113370 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.113344 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-486cd4-5758d6dfc9-v5nfj" event={"ID":"8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1","Type":"ContainerDied","Data":"157ad8f9be2f8b7ddb1b98c068fe70ac6e094661a9304c6e782528f4848f48e0"} Oct 30 07:19:54.113493 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.113411 4951 scope.go:122] "RemoveContainer" containerID="c0541eafaad5dae72820871097d23c1509ce657ef20fc4ee509f6dc2910abd58" Oct 30 07:19:54.343620 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.310233 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istio-token\") pod \"e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:19:54.343620 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.310331 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istiod-ca-cert\") pod \"e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:19:54.343620 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.310496 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-credential-socket\") pod \"e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:19:54.343620 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.310560 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-workload-certs\") pod \"e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:19:54.343620 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.310603 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istio-envoy\") pod \"e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:19:54.343620 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.310676 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-workload-socket\") pod \"e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:19:54.343620 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.310703 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istio-podinfo\") pod \"e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:19:54.343620 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.310748 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zfhw\" (UniqueName: \"kubernetes.io/projected/0c51720b-1e0b-4cb0-a1e2-08065c91acde-kube-api-access-8zfhw\") pod \"e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:19:54.343620 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.310890 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istio-data\") pod \"e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:19:54.343620 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.311393 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istio-data\") pod \"e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:19:54.343620 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.312529 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istiod-ca-cert\") pod \"e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:19:54.343620 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.312789 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-credential-socket\") pod \"e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:19:54.343620 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.313007 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-workload-certs\") pod \"e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:19:54.347322 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.346928 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-workload-socket\") pod \"e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:19:54.382221 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.382137 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istio-podinfo\") pod \"e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:19:54.413597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.387200 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istio-envoy\") pod \"e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:19:54.453621 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.453352 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-486cd4-5db6fd58-tg4wr"] Oct 30 07:19:54.485823 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.483276 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-486cd4-5db6fd58-tg4wr"] Oct 30 07:19:54.512715 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.511473 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zfhw\" (UniqueName: \"kubernetes.io/projected/0c51720b-1e0b-4cb0-a1e2-08065c91acde-kube-api-access-8zfhw\") pod \"e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:19:54.527649 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.525458 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istio-token\") pod \"e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:19:54.534750 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.534645 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-486cd4-5758d6dfc9-v5nfj"] Oct 30 07:19:54.552596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.549945 4951 patch_prober.go:29] interesting pod/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 07:19:54.552596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.550024 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" podUID="011a2449-9f58-427a-aa6e-89e4a911c2e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 07:19:54.552596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.552454 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-486cd4-5758d6dfc9-v5nfj"] Oct 30 07:19:54.610491 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.610005 4951 patch_prober.go:29] interesting pod/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 07:19:54.610491 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.610071 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" podUID="2898e6e0-5229-4c67-b3e3-8f2076284b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 07:19:54.678624 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:54.675903 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:19:55.083979 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:55.083947 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx"] Oct 30 07:19:55.085511 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:19:55.085475 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c51720b_1e0b_4cb0_a1e2_08065c91acde.slice/crio-ec546b376dcac1a961fa6fe4d1be3bda5229706e19df080abeb67cd9eccccc16 WatchSource:0}: Error finding container ec546b376dcac1a961fa6fe4d1be3bda5229706e19df080abeb67cd9eccccc16: Status 404 returned error can't find the container with id ec546b376dcac1a961fa6fe4d1be3bda5229706e19df080abeb67cd9eccccc16 Oct 30 07:19:55.088290 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:55.088256 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:19:55.088383 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:55.088328 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:19:55.088383 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:55.088360 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:19:55.127051 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:55.127017 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" event={"ID":"0c51720b-1e0b-4cb0-a1e2-08065c91acde","Type":"ContainerStarted","Data":"ec546b376dcac1a961fa6fe4d1be3bda5229706e19df080abeb67cd9eccccc16"} Oct 30 07:19:55.606646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:55.598880 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:19:55.648321 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:55.644905 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:19:55.648321 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:55.648258 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:19:55.656851 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:55.655537 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:19:55.656851 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:55.656709 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:19:55.659230 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:55.659162 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:19:55.786440 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:55.785082 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10fee9bf-b760-42ee-bd0c-93a44cee033e" path="/var/lib/kubelet/pods/10fee9bf-b760-42ee-bd0c-93a44cee033e/volumes" Oct 30 07:19:55.786440 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:55.785600 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1" path="/var/lib/kubelet/pods/8f94c9b9-ddc4-4a82-a3ce-5f9f78d920c1/volumes" Oct 30 07:19:55.786440 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:55.785921 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-794fc67c4c-z2hjz"] Oct 30 07:19:55.791679 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:55.789658 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-794fc67c4c-z2hjz" Oct 30 07:19:55.812448 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:55.812414 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0cd8f0-dockercfg-pgbsb\"" Oct 30 07:19:55.813221 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:55.813192 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0cd8f0\"" Oct 30 07:19:55.814799 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:55.814773 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-794fc67c4c-z2hjz"] Oct 30 07:19:55.850831 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:55.850799 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7npg\" (UniqueName: \"kubernetes.io/projected/4a995882-9616-44c9-8f3c-2d6615d7bbde-kube-api-access-r7npg\") pod \"payload-pre-processing-e2e-ipp-a-0cd8f0-794fc67c4c-z2hjz\" (UID: \"4a995882-9616-44c9-8f3c-2d6615d7bbde\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-794fc67c4c-z2hjz" Oct 30 07:19:55.854874 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:55.854843 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a995882-9616-44c9-8f3c-2d6615d7bbde-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0cd8f0-794fc67c4c-z2hjz\" (UID: \"4a995882-9616-44c9-8f3c-2d6615d7bbde\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-794fc67c4c-z2hjz" Oct 30 07:19:55.988648 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:55.976460 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-6bbb96bf6d-wwc5q"] Oct 30 07:19:55.988648 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:55.985549 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a995882-9616-44c9-8f3c-2d6615d7bbde-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0cd8f0-794fc67c4c-z2hjz\" (UID: \"4a995882-9616-44c9-8f3c-2d6615d7bbde\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-794fc67c4c-z2hjz" Oct 30 07:19:55.988648 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:55.986633 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a995882-9616-44c9-8f3c-2d6615d7bbde-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0cd8f0-794fc67c4c-z2hjz\" (UID: \"4a995882-9616-44c9-8f3c-2d6615d7bbde\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-794fc67c4c-z2hjz" Oct 30 07:19:55.988648 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:55.986855 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7npg\" (UniqueName: \"kubernetes.io/projected/4a995882-9616-44c9-8f3c-2d6615d7bbde-kube-api-access-r7npg\") pod \"payload-pre-processing-e2e-ipp-a-0cd8f0-794fc67c4c-z2hjz\" (UID: \"4a995882-9616-44c9-8f3c-2d6615d7bbde\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-794fc67c4c-z2hjz" Oct 30 07:19:56.034607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:56.030970 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-6bbb96bf6d-wwc5q"] Oct 30 07:19:56.034607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:56.031111 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-6bbb96bf6d-wwc5q" Oct 30 07:19:56.056664 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:56.053390 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7npg\" (UniqueName: \"kubernetes.io/projected/4a995882-9616-44c9-8f3c-2d6615d7bbde-kube-api-access-r7npg\") pod \"payload-pre-processing-e2e-ipp-a-0cd8f0-794fc67c4c-z2hjz\" (UID: \"4a995882-9616-44c9-8f3c-2d6615d7bbde\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-794fc67c4c-z2hjz" Oct 30 07:19:56.094991 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:56.094936 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfrsv\" (UniqueName: \"kubernetes.io/projected/d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450-kube-api-access-zfrsv\") pod \"payload-processing-e2e-ipp-a-0cd8f0-6bbb96bf6d-wwc5q\" (UID: \"d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-6bbb96bf6d-wwc5q" Oct 30 07:19:56.095450 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:56.095222 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0cd8f0-6bbb96bf6d-wwc5q\" (UID: \"d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-6bbb96bf6d-wwc5q" Oct 30 07:19:56.153283 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:56.153247 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-794fc67c4c-z2hjz" Oct 30 07:19:56.169524 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:56.169490 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" event={"ID":"0c51720b-1e0b-4cb0-a1e2-08065c91acde","Type":"ContainerStarted","Data":"b34987ce0a1fefa04c782ce4dfe7aaaae0d8f38b4711817ea271e2d96088e65f"} Oct 30 07:19:56.200338 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:56.200301 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0cd8f0-6bbb96bf6d-wwc5q\" (UID: \"d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-6bbb96bf6d-wwc5q" Oct 30 07:19:56.200661 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:56.200634 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfrsv\" (UniqueName: \"kubernetes.io/projected/d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450-kube-api-access-zfrsv\") pod \"payload-processing-e2e-ipp-a-0cd8f0-6bbb96bf6d-wwc5q\" (UID: \"d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-6bbb96bf6d-wwc5q" Oct 30 07:19:56.201730 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:56.201705 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0cd8f0-6bbb96bf6d-wwc5q\" (UID: \"d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-6bbb96bf6d-wwc5q" Oct 30 07:19:56.222187 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:56.222134 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" podStartSLOduration=3.222115413 podStartE2EDuration="3.222115413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:19:56.216943709 +0000 UTC m=+1402.559991709" watchObservedRunningTime="2026-10-30 07:19:56.222115413 +0000 UTC m=+1402.565163380" Oct 30 07:19:56.229537 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:56.229507 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfrsv\" (UniqueName: \"kubernetes.io/projected/d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450-kube-api-access-zfrsv\") pod \"payload-processing-e2e-ipp-a-0cd8f0-6bbb96bf6d-wwc5q\" (UID: \"d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-6bbb96bf6d-wwc5q" Oct 30 07:19:56.452042 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:56.450861 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-6bbb96bf6d-wwc5q" Oct 30 07:19:56.509221 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:56.509186 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-794fc67c4c-z2hjz"] Oct 30 07:19:56.510508 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:19:56.510482 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a995882_9616_44c9_8f3c_2d6615d7bbde.slice/crio-53bff5b202c5943bbf492f02a96adcd88a7eafb6c47e001d4af33da41756dc8e WatchSource:0}: Error finding container 53bff5b202c5943bbf492f02a96adcd88a7eafb6c47e001d4af33da41756dc8e: Status 404 returned error can't find the container with id 53bff5b202c5943bbf492f02a96adcd88a7eafb6c47e001d4af33da41756dc8e Oct 30 07:19:56.681296 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:56.681261 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:19:56.686663 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:56.686640 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:19:56.791855 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:56.791831 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-6bbb96bf6d-wwc5q"] Oct 30 07:19:56.794000 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:19:56.793970 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2fda5b5_fcf9_4377_8b53_4e9ffa0a6450.slice/crio-06c9243981fff5efbe24f51001c0ff8479f3d468a84fa3f5f1054f2125c137b7 WatchSource:0}: Error finding container 06c9243981fff5efbe24f51001c0ff8479f3d468a84fa3f5f1054f2125c137b7: Status 404 returned error can't find the container with id 06c9243981fff5efbe24f51001c0ff8479f3d468a84fa3f5f1054f2125c137b7 Oct 30 07:19:57.175936 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:57.175842 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-6bbb96bf6d-wwc5q" event={"ID":"d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450","Type":"ContainerStarted","Data":"06c9243981fff5efbe24f51001c0ff8479f3d468a84fa3f5f1054f2125c137b7"} Oct 30 07:19:57.180905 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:57.180867 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-794fc67c4c-z2hjz" event={"ID":"4a995882-9616-44c9-8f3c-2d6615d7bbde","Type":"ContainerStarted","Data":"53bff5b202c5943bbf492f02a96adcd88a7eafb6c47e001d4af33da41756dc8e"} Oct 30 07:19:57.182138 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:57.182113 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:19:57.183235 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:57.183214 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:19:59.200651 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:59.200531 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-6bbb96bf6d-wwc5q" event={"ID":"d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450","Type":"ContainerStarted","Data":"1fb10342388d7c014b375457cc3f79d06ddf3b48b57ea64a8af53f0211b6bb76"} Oct 30 07:19:59.201093 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:59.200779 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-6bbb96bf6d-wwc5q" Oct 30 07:19:59.202478 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:59.202443 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-794fc67c4c-z2hjz" event={"ID":"4a995882-9616-44c9-8f3c-2d6615d7bbde","Type":"ContainerStarted","Data":"cd75c1b7ea8ceef193e78de57f949a9d189c471cb866fa82ccafc4fdf3c86d70"} Oct 30 07:19:59.202716 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:59.202698 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-794fc67c4c-z2hjz" Oct 30 07:19:59.244167 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:59.244103 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-6bbb96bf6d-wwc5q" podStartSLOduration=2.252739486 podStartE2EDuration="4.244084342s" totalImagesPullingTime="1.991344856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:19:59.241128492 +0000 UTC m=+1405.584176457" watchObservedRunningTime="2026-10-30 07:19:59.244084342 +0000 UTC m=+1405.587132309" Oct 30 07:19:59.283123 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:19:59.283053 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-794fc67c4c-z2hjz" podStartSLOduration=2.26998419 podStartE2EDuration="4.283035793s" totalImagesPullingTime="2.013051603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:19:59.281115309 +0000 UTC m=+1405.624163296" watchObservedRunningTime="2026-10-30 07:19:59.283035793 +0000 UTC m=+1405.626083759" Oct 30 07:20:00.312901 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:00.312863 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72"] Oct 30 07:20:00.313871 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:00.313837 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" podUID="42ea0497-3b5f-4c33-a53a-bd636a4379ba" containerName="istio-proxy" containerID="cri-o://0c79da0deb4539c1b8607b87e63d6be137320dc2ffe3ebe69a76feaed8c732f0" gracePeriod=30 Oct 30 07:20:00.328132 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:00.327954 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs"] Oct 30 07:20:00.329005 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:00.328227 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" podUID="510f006a-0511-4993-8fdc-9e5ee1d6f048" containerName="istio-proxy" containerID="cri-o://9a966c0166ca4ddcf117e4671e93a491478852d7592fccdbe9ed7185f51d2532" gracePeriod=30 Oct 30 07:20:00.336955 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:00.333133 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv"] Oct 30 07:20:00.336955 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:00.333373 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" podUID="a97844ee-24e2-4039-80d1-559b7b9f03bd" containerName="istio-proxy" containerID="cri-o://5cf2f8c236380edc6c9a9ed7ecf088dc7cae92e333ecbc9a43f8f14db252293b" gracePeriod=30 Oct 30 07:20:00.356145 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:00.342527 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7"] Oct 30 07:20:00.356145 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:00.342861 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" podUID="2f0d4540-46c9-4f4f-85d2-c9ee3880efa0" containerName="istio-proxy" containerID="cri-o://18c5c96db1706f6dd628297320cbad8a4587de62addb17a15014f99a449c2d90" gracePeriod=30 Oct 30 07:20:00.862583 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:00.862466 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-438855-5cf54f6f44-wbkqt"] Oct 30 07:20:00.885630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:00.885552 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-438855-5cf54f6f44-wbkqt"] Oct 30 07:20:00.885986 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:00.885967 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-5cf54f6f44-wbkqt" Oct 30 07:20:00.914344 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:00.914313 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-438855-dockercfg-65zgz\"" Oct 30 07:20:00.931872 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:00.917662 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-438855\"" Oct 30 07:20:01.162594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:01.146427 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/923bebff-1573-4dbc-997c-24a416579f99-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-438855-5cf54f6f44-wbkqt\" (UID: \"923bebff-1573-4dbc-997c-24a416579f99\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-5cf54f6f44-wbkqt" Oct 30 07:20:01.163035 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:01.163003 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgg6g\" (UniqueName: \"kubernetes.io/projected/923bebff-1573-4dbc-997c-24a416579f99-kube-api-access-tgg6g\") pod \"payload-processing-e2e-ipp-a-438855-5cf54f6f44-wbkqt\" (UID: \"923bebff-1573-4dbc-997c-24a416579f99\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-5cf54f6f44-wbkqt" Oct 30 07:20:01.402647 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:01.333237 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/923bebff-1573-4dbc-997c-24a416579f99-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-438855-5cf54f6f44-wbkqt\" (UID: \"923bebff-1573-4dbc-997c-24a416579f99\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-5cf54f6f44-wbkqt" Oct 30 07:20:01.402647 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:01.333312 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgg6g\" (UniqueName: \"kubernetes.io/projected/923bebff-1573-4dbc-997c-24a416579f99-kube-api-access-tgg6g\") pod \"payload-processing-e2e-ipp-a-438855-5cf54f6f44-wbkqt\" (UID: \"923bebff-1573-4dbc-997c-24a416579f99\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-5cf54f6f44-wbkqt" Oct 30 07:20:01.402647 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:01.355193 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/923bebff-1573-4dbc-997c-24a416579f99-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-438855-5cf54f6f44-wbkqt\" (UID: \"923bebff-1573-4dbc-997c-24a416579f99\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-5cf54f6f44-wbkqt" Oct 30 07:20:01.447742 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:01.447295 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgg6g\" (UniqueName: \"kubernetes.io/projected/923bebff-1573-4dbc-997c-24a416579f99-kube-api-access-tgg6g\") pod \"payload-processing-e2e-ipp-a-438855-5cf54f6f44-wbkqt\" (UID: \"923bebff-1573-4dbc-997c-24a416579f99\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-5cf54f6f44-wbkqt" Oct 30 07:20:01.511182 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:01.511145 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-5cf54f6f44-wbkqt" Oct 30 07:20:01.833804 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:20:01.833766 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod923bebff_1573_4dbc_997c_24a416579f99.slice/crio-2ed9ccd07f09557bbc5ea988cb65d41fa9fc2bc77766e04205d41c89f5eb53ff WatchSource:0}: Error finding container 2ed9ccd07f09557bbc5ea988cb65d41fa9fc2bc77766e04205d41c89f5eb53ff: Status 404 returned error can't find the container with id 2ed9ccd07f09557bbc5ea988cb65d41fa9fc2bc77766e04205d41c89f5eb53ff Oct 30 07:20:01.834123 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:01.834093 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-438855-5cf54f6f44-wbkqt"] Oct 30 07:20:02.479729 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:02.479687 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-5cf54f6f44-wbkqt" event={"ID":"923bebff-1573-4dbc-997c-24a416579f99","Type":"ContainerStarted","Data":"4837b93a949ed6b52f3ec1891b238040b1a31bc75367f0ca817fffe9f949625e"} Oct 30 07:20:02.480188 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:02.479750 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-5cf54f6f44-wbkqt" event={"ID":"923bebff-1573-4dbc-997c-24a416579f99","Type":"ContainerStarted","Data":"2ed9ccd07f09557bbc5ea988cb65d41fa9fc2bc77766e04205d41c89f5eb53ff"} Oct 30 07:20:02.480188 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:02.479956 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-5cf54f6f44-wbkqt" Oct 30 07:20:02.526003 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:02.525940 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-5cf54f6f44-wbkqt" podStartSLOduration=2.525921228 podStartE2EDuration="2.525921228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:20:02.520660958 +0000 UTC m=+1408.863708942" watchObservedRunningTime="2026-10-30 07:20:02.525921228 +0000 UTC m=+1408.868969202" Oct 30 07:20:04.247161 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:04.247119 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz"] Oct 30 07:20:04.247718 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:04.247354 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz" podUID="b2076f66-8e90-4c87-ab08-6d4b5e3280b6" containerName="maas-api" containerID="cri-o://d0730eb23d2e0d6d54ff18e863fc16a0109b32f0bcecac60e5d38054771ca55e" gracePeriod=30 Oct 30 07:20:04.569565 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:04.569517 4951 generic.go:356] "Generic (PLEG): container finished" podID="b2076f66-8e90-4c87-ab08-6d4b5e3280b6" containerID="d0730eb23d2e0d6d54ff18e863fc16a0109b32f0bcecac60e5d38054771ca55e" exitCode=0 Oct 30 07:20:04.569565 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:04.569660 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz" event={"ID":"b2076f66-8e90-4c87-ab08-6d4b5e3280b6","Type":"ContainerDied","Data":"d0730eb23d2e0d6d54ff18e863fc16a0109b32f0bcecac60e5d38054771ca55e"} Oct 30 07:20:04.640247 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:04.639820 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz" Oct 30 07:20:04.668628 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:04.654282 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2076f66-8e90-4c87-ab08-6d4b5e3280b6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2076f66-8e90-4c87-ab08-6d4b5e3280b6-maas-api-tls\") pod \"b2076f66-8e90-4c87-ab08-6d4b5e3280b6\" (UID: \"b2076f66-8e90-4c87-ab08-6d4b5e3280b6\") " Oct 30 07:20:04.668628 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:04.654490 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2076f66-8e90-4c87-ab08-6d4b5e3280b6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2076f66-8e90-4c87-ab08-6d4b5e3280b6-openshift-service-ca\") pod \"b2076f66-8e90-4c87-ab08-6d4b5e3280b6\" (UID: \"b2076f66-8e90-4c87-ab08-6d4b5e3280b6\") " Oct 30 07:20:04.668628 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:04.654538 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2076f66-8e90-4c87-ab08-6d4b5e3280b6-kube-api-access-gwwd9\" (UniqueName: \"kubernetes.io/projected/b2076f66-8e90-4c87-ab08-6d4b5e3280b6-kube-api-access-gwwd9\") pod \"b2076f66-8e90-4c87-ab08-6d4b5e3280b6\" (UID: \"b2076f66-8e90-4c87-ab08-6d4b5e3280b6\") " Oct 30 07:20:04.668628 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:04.654825 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2076f66-8e90-4c87-ab08-6d4b5e3280b6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2076f66-8e90-4c87-ab08-6d4b5e3280b6-metrics-certs\") pod \"b2076f66-8e90-4c87-ab08-6d4b5e3280b6\" (UID: \"b2076f66-8e90-4c87-ab08-6d4b5e3280b6\") " Oct 30 07:20:04.668628 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:04.660805 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2076f66-8e90-4c87-ab08-6d4b5e3280b6-openshift-service-ca" pod "b2076f66-8e90-4c87-ab08-6d4b5e3280b6" (UID: "b2076f66-8e90-4c87-ab08-6d4b5e3280b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:20:04.721212 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:04.721095 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2076f66-8e90-4c87-ab08-6d4b5e3280b6-maas-api-tls" pod "b2076f66-8e90-4c87-ab08-6d4b5e3280b6" (UID: "b2076f66-8e90-4c87-ab08-6d4b5e3280b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:20:04.740862 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:04.737170 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2076f66-8e90-4c87-ab08-6d4b5e3280b6-kube-api-access-gwwd9" pod "b2076f66-8e90-4c87-ab08-6d4b5e3280b6" (UID: "b2076f66-8e90-4c87-ab08-6d4b5e3280b6"). InnerVolumeSpecName "kube-api-access-gwwd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:20:04.740862 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:04.737634 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2076f66-8e90-4c87-ab08-6d4b5e3280b6-metrics-certs" pod "b2076f66-8e90-4c87-ab08-6d4b5e3280b6" (UID: "b2076f66-8e90-4c87-ab08-6d4b5e3280b6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:20:04.884780 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:04.865882 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2076f66-8e90-4c87-ab08-6d4b5e3280b6-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:04.884780 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:04.865914 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwwd9\" (UniqueName: \"kubernetes.io/projected/b2076f66-8e90-4c87-ab08-6d4b5e3280b6-kube-api-access-gwwd9\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:04.884780 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:04.865929 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2076f66-8e90-4c87-ab08-6d4b5e3280b6-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:04.884780 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:04.865943 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2076f66-8e90-4c87-ab08-6d4b5e3280b6-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.590254 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.589792 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz" event={"ID":"b2076f66-8e90-4c87-ab08-6d4b5e3280b6","Type":"ContainerDied","Data":"8374e74ad159eca3a94d9dea4867e9b76566cf4fa21e5ac0acb685dec6b3a503"} Oct 30 07:20:05.590254 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.589872 4951 scope.go:122] "RemoveContainer" containerID="d0730eb23d2e0d6d54ff18e863fc16a0109b32f0bcecac60e5d38054771ca55e" Oct 30 07:20:05.590254 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.590047 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz" Oct 30 07:20:05.593507 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.593380 4951 generic.go:356] "Generic (PLEG): container finished" podID="42ea0497-3b5f-4c33-a53a-bd636a4379ba" containerID="0c79da0deb4539c1b8607b87e63d6be137320dc2ffe3ebe69a76feaed8c732f0" exitCode=0 Oct 30 07:20:05.593507 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.593486 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" event={"ID":"42ea0497-3b5f-4c33-a53a-bd636a4379ba","Type":"ContainerDied","Data":"0c79da0deb4539c1b8607b87e63d6be137320dc2ffe3ebe69a76feaed8c732f0"} Oct 30 07:20:05.596088 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.595255 4951 generic.go:356] "Generic (PLEG): container finished" podID="510f006a-0511-4993-8fdc-9e5ee1d6f048" containerID="9a966c0166ca4ddcf117e4671e93a491478852d7592fccdbe9ed7185f51d2532" exitCode=0 Oct 30 07:20:05.596088 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.595348 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" event={"ID":"510f006a-0511-4993-8fdc-9e5ee1d6f048","Type":"ContainerDied","Data":"9a966c0166ca4ddcf117e4671e93a491478852d7592fccdbe9ed7185f51d2532"} Oct 30 07:20:05.597463 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.597428 4951 generic.go:356] "Generic (PLEG): container finished" podID="a97844ee-24e2-4039-80d1-559b7b9f03bd" containerID="5cf2f8c236380edc6c9a9ed7ecf088dc7cae92e333ecbc9a43f8f14db252293b" exitCode=0 Oct 30 07:20:05.597660 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.597473 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" event={"ID":"a97844ee-24e2-4039-80d1-559b7b9f03bd","Type":"ContainerDied","Data":"5cf2f8c236380edc6c9a9ed7ecf088dc7cae92e333ecbc9a43f8f14db252293b"} Oct 30 07:20:05.599320 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.599299 4951 generic.go:356] "Generic (PLEG): container finished" podID="2f0d4540-46c9-4f4f-85d2-c9ee3880efa0" containerID="18c5c96db1706f6dd628297320cbad8a4587de62addb17a15014f99a449c2d90" exitCode=0 Oct 30 07:20:05.599425 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.599351 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" event={"ID":"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0","Type":"ContainerDied","Data":"18c5c96db1706f6dd628297320cbad8a4587de62addb17a15014f99a449c2d90"} Oct 30 07:20:05.606456 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.606438 4951 scope.go:122] "RemoveContainer" containerID="17f52bcda66d0ed800ec1c3ef785437e87ec0ec6dae43053a48e1cbce9f9de28" Oct 30 07:20:05.645867 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.645840 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz"] Oct 30 07:20:05.659935 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.659869 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4c43f7-77b8bff858-jr6qz"] Oct 30 07:20:05.689618 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.689564 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:20:05.759777 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.759452 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:20:05.775334 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.775308 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2076f66-8e90-4c87-ab08-6d4b5e3280b6" path="/var/lib/kubelet/pods/b2076f66-8e90-4c87-ab08-6d4b5e3280b6/volumes" Oct 30 07:20:05.778772 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.778728 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:20:05.780472 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.780411 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:20:05.793864 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.793821 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a97844ee-24e2-4039-80d1-559b7b9f03bd-kube-api-access-kppmq\" (UniqueName: \"kubernetes.io/projected/a97844ee-24e2-4039-80d1-559b7b9f03bd-kube-api-access-kppmq\") pod \"a97844ee-24e2-4039-80d1-559b7b9f03bd\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " Oct 30 07:20:05.794078 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.794048 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/510f006a-0511-4993-8fdc-9e5ee1d6f048-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/510f006a-0511-4993-8fdc-9e5ee1d6f048-istiod-ca-cert\") pod \"510f006a-0511-4993-8fdc-9e5ee1d6f048\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " Oct 30 07:20:05.794194 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.794173 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-credential-socket\") pod \"a97844ee-24e2-4039-80d1-559b7b9f03bd\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " Oct 30 07:20:05.794263 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.794222 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a97844ee-24e2-4039-80d1-559b7b9f03bd-istio-token\" (UniqueName: \"kubernetes.io/projected/a97844ee-24e2-4039-80d1-559b7b9f03bd-istio-token\") pod \"a97844ee-24e2-4039-80d1-559b7b9f03bd\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " Oct 30 07:20:05.794322 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.794265 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/510f006a-0511-4993-8fdc-9e5ee1d6f048-istio-token\" (UniqueName: \"kubernetes.io/projected/510f006a-0511-4993-8fdc-9e5ee1d6f048-istio-token\") pod \"510f006a-0511-4993-8fdc-9e5ee1d6f048\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " Oct 30 07:20:05.794322 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.794299 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-workload-socket\") pod \"510f006a-0511-4993-8fdc-9e5ee1d6f048\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " Oct 30 07:20:05.794416 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.794340 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istio-token\" (UniqueName: \"kubernetes.io/projected/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istio-token\") pod \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " Oct 30 07:20:05.794474 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.794453 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/510f006a-0511-4993-8fdc-9e5ee1d6f048-kube-api-access-8td8h\" (UniqueName: \"kubernetes.io/projected/510f006a-0511-4993-8fdc-9e5ee1d6f048-kube-api-access-8td8h\") pod \"510f006a-0511-4993-8fdc-9e5ee1d6f048\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " Oct 30 07:20:05.794526 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.794498 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istiod-ca-cert\") pod \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " Oct 30 07:20:05.794687 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.794664 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istio-token\" (UniqueName: \"kubernetes.io/projected/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istio-token\") pod \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " Oct 30 07:20:05.794957 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.794927 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-credential-socket" pod "a97844ee-24e2-4039-80d1-559b7b9f03bd" (UID: "a97844ee-24e2-4039-80d1-559b7b9f03bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:20:05.795033 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.794510 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/510f006a-0511-4993-8fdc-9e5ee1d6f048-istiod-ca-cert" pod "510f006a-0511-4993-8fdc-9e5ee1d6f048" (UID: "510f006a-0511-4993-8fdc-9e5ee1d6f048"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:20:05.795236 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.795212 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istiod-ca-cert" pod "2f0d4540-46c9-4f4f-85d2-c9ee3880efa0" (UID: "2f0d4540-46c9-4f4f-85d2-c9ee3880efa0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:20:05.795308 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.795224 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/510f006a-0511-4993-8fdc-9e5ee1d6f048-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/510f006a-0511-4993-8fdc-9e5ee1d6f048-istio-podinfo\") pod \"510f006a-0511-4993-8fdc-9e5ee1d6f048\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " Oct 30 07:20:05.795308 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.795278 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-workload-socket\") pod \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " Oct 30 07:20:05.795415 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.795318 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istio-envoy\") pod \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " Oct 30 07:20:05.795498 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.795473 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42ea0497-3b5f-4c33-a53a-bd636a4379ba-kube-api-access-w96pb\" (UniqueName: \"kubernetes.io/projected/42ea0497-3b5f-4c33-a53a-bd636a4379ba-kube-api-access-w96pb\") pod \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " Oct 30 07:20:05.795742 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.795714 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-workload-socket\") pod \"a97844ee-24e2-4039-80d1-559b7b9f03bd\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " Oct 30 07:20:05.795836 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.795806 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istio-podinfo\") pod \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " Oct 30 07:20:05.795898 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.795848 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istio-data\") pod \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " Oct 30 07:20:05.795956 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.795904 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-workload-certs\") pod \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " Oct 30 07:20:05.796020 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.795973 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-istio-data\") pod \"a97844ee-24e2-4039-80d1-559b7b9f03bd\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " Oct 30 07:20:05.796793 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.796444 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-workload-socket" pod "a97844ee-24e2-4039-80d1-559b7b9f03bd" (UID: "a97844ee-24e2-4039-80d1-559b7b9f03bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:20:05.796793 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.796456 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-workload-socket" pod "510f006a-0511-4993-8fdc-9e5ee1d6f048" (UID: "510f006a-0511-4993-8fdc-9e5ee1d6f048"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:20:05.796793 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.796014 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-workload-certs\") pod \"a97844ee-24e2-4039-80d1-559b7b9f03bd\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " Oct 30 07:20:05.796793 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.796533 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istio-envoy\") pod \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " Oct 30 07:20:05.796793 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.796564 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a97844ee-24e2-4039-80d1-559b7b9f03bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a97844ee-24e2-4039-80d1-559b7b9f03bd-istiod-ca-cert\") pod \"a97844ee-24e2-4039-80d1-559b7b9f03bd\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " Oct 30 07:20:05.796793 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.796631 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-istio-envoy\") pod \"a97844ee-24e2-4039-80d1-559b7b9f03bd\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " Oct 30 07:20:05.796793 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.796678 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istiod-ca-cert\") pod \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " Oct 30 07:20:05.796793 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.796728 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-kube-api-access-bn52f\" (UniqueName: \"kubernetes.io/projected/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-kube-api-access-bn52f\") pod \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " Oct 30 07:20:05.797858 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.797672 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istio-data" pod "2f0d4540-46c9-4f4f-85d2-c9ee3880efa0" (UID: "2f0d4540-46c9-4f4f-85d2-c9ee3880efa0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:20:05.798027 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.797809 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-workload-certs" pod "a97844ee-24e2-4039-80d1-559b7b9f03bd" (UID: "a97844ee-24e2-4039-80d1-559b7b9f03bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:20:05.798027 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.797860 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-workload-socket" pod "42ea0497-3b5f-4c33-a53a-bd636a4379ba" (UID: "42ea0497-3b5f-4c33-a53a-bd636a4379ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:20:05.798027 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.797882 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-workload-certs" pod "42ea0497-3b5f-4c33-a53a-bd636a4379ba" (UID: "42ea0497-3b5f-4c33-a53a-bd636a4379ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:20:05.798219 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.798075 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-istio-data" pod "a97844ee-24e2-4039-80d1-559b7b9f03bd" (UID: "a97844ee-24e2-4039-80d1-559b7b9f03bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:20:05.798219 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.798124 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-workload-certs\") pod \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " Oct 30 07:20:05.798219 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.798166 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-istio-data\") pod \"510f006a-0511-4993-8fdc-9e5ee1d6f048\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " Oct 30 07:20:05.798402 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.798221 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-istio-envoy\") pod \"510f006a-0511-4993-8fdc-9e5ee1d6f048\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " Oct 30 07:20:05.798402 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.798257 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-credential-socket\") pod \"510f006a-0511-4993-8fdc-9e5ee1d6f048\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " Oct 30 07:20:05.798402 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.798294 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a97844ee-24e2-4039-80d1-559b7b9f03bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a97844ee-24e2-4039-80d1-559b7b9f03bd-istio-podinfo\") pod \"a97844ee-24e2-4039-80d1-559b7b9f03bd\" (UID: \"a97844ee-24e2-4039-80d1-559b7b9f03bd\") " Oct 30 07:20:05.798402 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.798331 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istio-data\") pod \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " Oct 30 07:20:05.798402 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.798369 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-workload-socket\") pod \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " Oct 30 07:20:05.798682 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.798466 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-credential-socket\") pod \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " Oct 30 07:20:05.798682 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.798536 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-credential-socket\") pod \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\" (UID: \"42ea0497-3b5f-4c33-a53a-bd636a4379ba\") " Oct 30 07:20:05.798682 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.798609 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istio-podinfo\") pod \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\" (UID: \"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0\") " Oct 30 07:20:05.798682 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.798660 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-workload-certs\") pod \"510f006a-0511-4993-8fdc-9e5ee1d6f048\" (UID: \"510f006a-0511-4993-8fdc-9e5ee1d6f048\") " Oct 30 07:20:05.800173 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.800146 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-workload-certs" pod "510f006a-0511-4993-8fdc-9e5ee1d6f048" (UID: "510f006a-0511-4993-8fdc-9e5ee1d6f048"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:20:05.800268 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.800239 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istio-token" pod "42ea0497-3b5f-4c33-a53a-bd636a4379ba" (UID: "42ea0497-3b5f-4c33-a53a-bd636a4379ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:20:05.800375 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.800353 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-workload-certs" pod "2f0d4540-46c9-4f4f-85d2-c9ee3880efa0" (UID: "2f0d4540-46c9-4f4f-85d2-c9ee3880efa0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:20:05.800375 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.800365 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istio-podinfo" pod "42ea0497-3b5f-4c33-a53a-bd636a4379ba" (UID: "42ea0497-3b5f-4c33-a53a-bd636a4379ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:20:05.800491 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.800382 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istio-envoy" pod "2f0d4540-46c9-4f4f-85d2-c9ee3880efa0" (UID: "2f0d4540-46c9-4f4f-85d2-c9ee3880efa0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:20:05.801193 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.801165 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510f006a-0511-4993-8fdc-9e5ee1d6f048-istio-token" pod "510f006a-0511-4993-8fdc-9e5ee1d6f048" (UID: "510f006a-0511-4993-8fdc-9e5ee1d6f048"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:20:05.801300 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.801228 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a97844ee-24e2-4039-80d1-559b7b9f03bd-istio-token" pod "a97844ee-24e2-4039-80d1-559b7b9f03bd" (UID: "a97844ee-24e2-4039-80d1-559b7b9f03bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:20:05.801300 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.801272 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istio-token" pod "2f0d4540-46c9-4f4f-85d2-c9ee3880efa0" (UID: "2f0d4540-46c9-4f4f-85d2-c9ee3880efa0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:20:05.801415 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.801343 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-istio-data" pod "510f006a-0511-4993-8fdc-9e5ee1d6f048" (UID: "510f006a-0511-4993-8fdc-9e5ee1d6f048"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:20:05.802245 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.802221 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.802341 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.802246 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/510f006a-0511-4993-8fdc-9e5ee1d6f048-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.802341 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.802260 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.802341 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.802274 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a97844ee-24e2-4039-80d1-559b7b9f03bd-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.802341 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.802287 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/510f006a-0511-4993-8fdc-9e5ee1d6f048-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.802341 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.802300 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.802341 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.802313 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.802341 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.802328 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.802341 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.802340 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.802780 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.802352 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.802780 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.802220 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a97844ee-24e2-4039-80d1-559b7b9f03bd-kube-api-access-kppmq" pod "a97844ee-24e2-4039-80d1-559b7b9f03bd" (UID: "a97844ee-24e2-4039-80d1-559b7b9f03bd"). InnerVolumeSpecName "kube-api-access-kppmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:20:05.802780 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.802455 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.802780 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.802471 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.802780 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.802473 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-workload-socket" pod "2f0d4540-46c9-4f4f-85d2-c9ee3880efa0" (UID: "2f0d4540-46c9-4f4f-85d2-c9ee3880efa0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:20:05.802780 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.802484 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.802780 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.802499 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.802780 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.802512 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.802780 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.802526 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.802780 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.802538 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.802780 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.802551 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.802780 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.802564 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.802780 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.802705 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-credential-socket" pod "510f006a-0511-4993-8fdc-9e5ee1d6f048" (UID: "510f006a-0511-4993-8fdc-9e5ee1d6f048"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:20:05.803786 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.802792 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-credential-socket" pod "42ea0497-3b5f-4c33-a53a-bd636a4379ba" (UID: "42ea0497-3b5f-4c33-a53a-bd636a4379ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:20:05.803786 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.802934 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istio-data" pod "42ea0497-3b5f-4c33-a53a-bd636a4379ba" (UID: "42ea0497-3b5f-4c33-a53a-bd636a4379ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:20:05.803786 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.803019 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-credential-socket" pod "2f0d4540-46c9-4f4f-85d2-c9ee3880efa0" (UID: "2f0d4540-46c9-4f4f-85d2-c9ee3880efa0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:20:05.803786 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.803369 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510f006a-0511-4993-8fdc-9e5ee1d6f048-kube-api-access-8td8h" pod "510f006a-0511-4993-8fdc-9e5ee1d6f048" (UID: "510f006a-0511-4993-8fdc-9e5ee1d6f048"). InnerVolumeSpecName "kube-api-access-8td8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:20:05.805468 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.805440 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a97844ee-24e2-4039-80d1-559b7b9f03bd-istio-podinfo" pod "a97844ee-24e2-4039-80d1-559b7b9f03bd" (UID: "a97844ee-24e2-4039-80d1-559b7b9f03bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:20:05.805820 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.805797 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a97844ee-24e2-4039-80d1-559b7b9f03bd-istiod-ca-cert" pod "a97844ee-24e2-4039-80d1-559b7b9f03bd" (UID: "a97844ee-24e2-4039-80d1-559b7b9f03bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:20:05.805888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.805837 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istiod-ca-cert" pod "42ea0497-3b5f-4c33-a53a-bd636a4379ba" (UID: "42ea0497-3b5f-4c33-a53a-bd636a4379ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:20:05.807512 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.807485 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istio-podinfo" pod "2f0d4540-46c9-4f4f-85d2-c9ee3880efa0" (UID: "2f0d4540-46c9-4f4f-85d2-c9ee3880efa0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:20:05.808153 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.808131 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-istio-envoy" pod "510f006a-0511-4993-8fdc-9e5ee1d6f048" (UID: "510f006a-0511-4993-8fdc-9e5ee1d6f048"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:20:05.817613 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.809264 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-istio-envoy" pod "a97844ee-24e2-4039-80d1-559b7b9f03bd" (UID: "a97844ee-24e2-4039-80d1-559b7b9f03bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:20:05.817613 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.810999 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ea0497-3b5f-4c33-a53a-bd636a4379ba-kube-api-access-w96pb" pod "42ea0497-3b5f-4c33-a53a-bd636a4379ba" (UID: "42ea0497-3b5f-4c33-a53a-bd636a4379ba"). InnerVolumeSpecName "kube-api-access-w96pb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:20:05.817613 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.811106 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/510f006a-0511-4993-8fdc-9e5ee1d6f048-istio-podinfo" pod "510f006a-0511-4993-8fdc-9e5ee1d6f048" (UID: "510f006a-0511-4993-8fdc-9e5ee1d6f048"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:20:05.817613 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.811913 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-kube-api-access-bn52f" pod "2f0d4540-46c9-4f4f-85d2-c9ee3880efa0" (UID: "2f0d4540-46c9-4f4f-85d2-c9ee3880efa0"). InnerVolumeSpecName "kube-api-access-bn52f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:20:05.822041 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.822005 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istio-envoy" pod "42ea0497-3b5f-4c33-a53a-bd636a4379ba" (UID: "42ea0497-3b5f-4c33-a53a-bd636a4379ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:20:05.903833 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.903781 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/510f006a-0511-4993-8fdc-9e5ee1d6f048-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.903833 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.903815 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w96pb\" (UniqueName: \"kubernetes.io/projected/42ea0497-3b5f-4c33-a53a-bd636a4379ba-kube-api-access-w96pb\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.903833 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.903831 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.903833 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.903843 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a97844ee-24e2-4039-80d1-559b7b9f03bd-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.904255 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.903856 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a97844ee-24e2-4039-80d1-559b7b9f03bd-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.904255 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.903867 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.904255 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.903879 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn52f\" (UniqueName: \"kubernetes.io/projected/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-kube-api-access-bn52f\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.904255 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.903891 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.904255 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.903905 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/510f006a-0511-4993-8fdc-9e5ee1d6f048-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.904255 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.903918 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a97844ee-24e2-4039-80d1-559b7b9f03bd-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.904255 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.903930 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.904255 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.903943 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.904255 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.903956 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.904255 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.903970 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ea0497-3b5f-4c33-a53a-bd636a4379ba-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.904255 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.903984 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.904255 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.903997 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kppmq\" (UniqueName: \"kubernetes.io/projected/a97844ee-24e2-4039-80d1-559b7b9f03bd-kube-api-access-kppmq\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:05.904255 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:05.904005 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8td8h\" (UniqueName: \"kubernetes.io/projected/510f006a-0511-4993-8fdc-9e5ee1d6f048-kube-api-access-8td8h\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:06.607793 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:06.607709 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" Oct 30 07:20:06.608222 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:06.607706 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv" event={"ID":"a97844ee-24e2-4039-80d1-559b7b9f03bd","Type":"ContainerDied","Data":"3cd3314d02c6fec1bb748399916e1e8f5313e9731c2700e63b4c6ce8888a0868"} Oct 30 07:20:06.608222 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:06.607918 4951 scope.go:122] "RemoveContainer" containerID="5cf2f8c236380edc6c9a9ed7ecf088dc7cae92e333ecbc9a43f8f14db252293b" Oct 30 07:20:06.609639 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:06.609615 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" Oct 30 07:20:06.609639 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:06.609624 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7" event={"ID":"2f0d4540-46c9-4f4f-85d2-c9ee3880efa0","Type":"ContainerDied","Data":"2e5d12bf5f27d084b874ed610a25d85b26d84b3ed3fed2e36a6153c791ce0f0c"} Oct 30 07:20:06.612275 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:06.612259 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" Oct 30 07:20:06.612405 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:06.612267 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72" event={"ID":"42ea0497-3b5f-4c33-a53a-bd636a4379ba","Type":"ContainerDied","Data":"dac69c03db8173fd5b4fc35bc8d0f73af287a329bc606d54f688c1576f944f5a"} Oct 30 07:20:06.618162 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:06.618136 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" event={"ID":"510f006a-0511-4993-8fdc-9e5ee1d6f048","Type":"ContainerDied","Data":"5e60a7fca814722342bac51199fe72c136fc3fe4c02a520fb3ae9ea586bcc781"} Oct 30 07:20:06.618278 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:06.618191 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs" Oct 30 07:20:06.619948 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:06.619906 4951 scope.go:122] "RemoveContainer" containerID="18c5c96db1706f6dd628297320cbad8a4587de62addb17a15014f99a449c2d90" Oct 30 07:20:06.631222 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:06.631208 4951 scope.go:122] "RemoveContainer" containerID="0c79da0deb4539c1b8607b87e63d6be137320dc2ffe3ebe69a76feaed8c732f0" Oct 30 07:20:06.642516 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:06.642496 4951 scope.go:122] "RemoveContainer" containerID="9a966c0166ca4ddcf117e4671e93a491478852d7592fccdbe9ed7185f51d2532" Oct 30 07:20:06.674184 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:06.674157 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7"] Oct 30 07:20:06.686244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:06.686219 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-46zx7"] Oct 30 07:20:06.716874 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:06.716849 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72"] Oct 30 07:20:06.720012 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:06.719986 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pk72"] Oct 30 07:20:06.753653 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:06.753619 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv"] Oct 30 07:20:06.761429 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:06.761394 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9m8cv"] Oct 30 07:20:06.792905 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:06.792872 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs"] Oct 30 07:20:06.798589 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:06.798536 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-q85zs"] Oct 30 07:20:07.772534 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:07.772499 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f0d4540-46c9-4f4f-85d2-c9ee3880efa0" path="/var/lib/kubelet/pods/2f0d4540-46c9-4f4f-85d2-c9ee3880efa0/volumes" Oct 30 07:20:07.773055 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:07.773034 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42ea0497-3b5f-4c33-a53a-bd636a4379ba" path="/var/lib/kubelet/pods/42ea0497-3b5f-4c33-a53a-bd636a4379ba/volumes" Oct 30 07:20:07.773540 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:07.773523 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="510f006a-0511-4993-8fdc-9e5ee1d6f048" path="/var/lib/kubelet/pods/510f006a-0511-4993-8fdc-9e5ee1d6f048/volumes" Oct 30 07:20:07.774324 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:07.774308 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a97844ee-24e2-4039-80d1-559b7b9f03bd" path="/var/lib/kubelet/pods/a97844ee-24e2-4039-80d1-559b7b9f03bd/volumes" Oct 30 07:20:10.269757 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:10.269726 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-6bbb96bf6d-wwc5q" Oct 30 07:20:10.270626 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:10.270001 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-794fc67c4c-z2hjz" Oct 30 07:20:11.565277 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.565246 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l"] Oct 30 07:20:11.566067 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.566044 4951 cpu_manager.go:414] "containerMap: removing container" podUID="42ea0497-3b5f-4c33-a53a-bd636a4379ba" containerName="istio-proxy" Oct 30 07:20:11.566178 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.566068 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42ea0497-3b5f-4c33-a53a-bd636a4379ba" containerName="istio-proxy" Oct 30 07:20:11.566178 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.566098 4951 cpu_manager.go:414] "containerMap: removing container" podUID="2f0d4540-46c9-4f4f-85d2-c9ee3880efa0" containerName="istio-proxy" Oct 30 07:20:11.566178 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.566106 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f0d4540-46c9-4f4f-85d2-c9ee3880efa0" containerName="istio-proxy" Oct 30 07:20:11.566178 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.566140 4951 cpu_manager.go:414] "containerMap: removing container" podUID="510f006a-0511-4993-8fdc-9e5ee1d6f048" containerName="istio-proxy" Oct 30 07:20:11.566178 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.566148 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510f006a-0511-4993-8fdc-9e5ee1d6f048" containerName="istio-proxy" Oct 30 07:20:11.566178 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.566160 4951 cpu_manager.go:414] "containerMap: removing container" podUID="42ea0497-3b5f-4c33-a53a-bd636a4379ba" containerName="istio-proxy" Oct 30 07:20:11.566178 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.566167 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42ea0497-3b5f-4c33-a53a-bd636a4379ba" containerName="istio-proxy" Oct 30 07:20:11.566178 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.566179 4951 cpu_manager.go:414] "containerMap: removing container" podUID="a97844ee-24e2-4039-80d1-559b7b9f03bd" containerName="istio-proxy" Oct 30 07:20:11.566563 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.566186 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a97844ee-24e2-4039-80d1-559b7b9f03bd" containerName="istio-proxy" Oct 30 07:20:11.566563 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.566197 4951 cpu_manager.go:414] "containerMap: removing container" podUID="b2076f66-8e90-4c87-ab08-6d4b5e3280b6" containerName="maas-api" Oct 30 07:20:11.566563 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.566204 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2076f66-8e90-4c87-ab08-6d4b5e3280b6" containerName="maas-api" Oct 30 07:20:11.566563 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.566383 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="42ea0497-3b5f-4c33-a53a-bd636a4379ba" containerName="istio-proxy" Oct 30 07:20:11.566563 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.566399 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f0d4540-46c9-4f4f-85d2-c9ee3880efa0" containerName="istio-proxy" Oct 30 07:20:11.566563 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.566412 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="510f006a-0511-4993-8fdc-9e5ee1d6f048" containerName="istio-proxy" Oct 30 07:20:11.566563 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.566424 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="42ea0497-3b5f-4c33-a53a-bd636a4379ba" containerName="istio-proxy" Oct 30 07:20:11.566563 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.566435 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="a97844ee-24e2-4039-80d1-559b7b9f03bd" containerName="istio-proxy" Oct 30 07:20:11.566563 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.566445 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2076f66-8e90-4c87-ab08-6d4b5e3280b6" containerName="maas-api" Oct 30 07:20:11.571073 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.571047 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:20:11.603906 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.603877 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-3d2270-openshift-default-dockercfg-r556l\"" Oct 30 07:20:11.610460 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.610433 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l"] Oct 30 07:20:11.727598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.720698 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-credential-socket\") pod \"e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:20:11.727598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.720764 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-workload-certs\") pod \"e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:20:11.727598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.720833 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istio-token\") pod \"e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:20:11.727598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.720896 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwc6q\" (UniqueName: \"kubernetes.io/projected/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-kube-api-access-pwc6q\") pod \"e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:20:11.727598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.720944 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-workload-socket\") pod \"e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:20:11.727598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.720971 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istio-envoy\") pod \"e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:20:11.727598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.721079 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istio-podinfo\") pod \"e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:20:11.727598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.721159 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istio-data\") pod \"e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:20:11.727598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.721206 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istiod-ca-cert\") pod \"e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:20:11.826666 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.826466 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-credential-socket\") pod \"e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:20:11.826666 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.826536 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-workload-certs\") pod \"e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:20:11.826666 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.826617 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istio-token\") pod \"e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:20:11.826963 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.826679 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwc6q\" (UniqueName: \"kubernetes.io/projected/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-kube-api-access-pwc6q\") pod \"e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:20:11.826963 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.826729 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-workload-socket\") pod \"e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:20:11.826963 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.826758 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istio-envoy\") pod \"e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:20:11.826963 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.826861 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istio-podinfo\") pod \"e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:20:11.826963 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.826930 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istio-data\") pod \"e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:20:11.827230 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.826979 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istiod-ca-cert\") pod \"e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:20:11.834592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.827892 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istiod-ca-cert\") pod \"e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:20:11.834592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.828187 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-credential-socket\") pod \"e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:20:11.834592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.828432 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-workload-certs\") pod \"e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:20:11.834592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.829277 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-workload-socket\") pod \"e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:20:11.834592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.830304 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istio-data\") pod \"e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:20:11.865701 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.854604 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istio-envoy\") pod \"e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:20:11.873890 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.868011 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istio-token\") pod \"e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:20:11.877595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.875207 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istio-podinfo\") pod \"e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:20:11.891381 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:11.889728 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwc6q\" (UniqueName: \"kubernetes.io/projected/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-kube-api-access-pwc6q\") pod \"e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:20:12.189024 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:12.188941 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:20:12.728895 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:12.728856 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l"] Oct 30 07:20:12.733354 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:12.733316 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:20:12.733481 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:12.733380 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:20:12.733481 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:12.733412 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:20:13.489763 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:13.489730 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-5cf54f6f44-wbkqt" Oct 30 07:20:13.686301 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:13.686261 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" event={"ID":"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b","Type":"ContainerStarted","Data":"893fe6868772fd31a178a7746a5974b29097011e48dbc7c66004b9e385ce56e5"} Oct 30 07:20:13.686476 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:13.686324 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" event={"ID":"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b","Type":"ContainerStarted","Data":"09dc38f5ea51b6b51df8f2faf57cbb5b7ddc62b4e367d4df9a78c6eaed6b6a19"} Oct 30 07:20:13.737160 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:13.737090 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" podStartSLOduration=2.73707053 podStartE2EDuration="2.73707053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:20:13.730171029 +0000 UTC m=+1420.073219004" watchObservedRunningTime="2026-10-30 07:20:13.73707053 +0000 UTC m=+1420.080118496" Oct 30 07:20:14.189206 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:14.189177 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:20:14.194949 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:14.194918 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:20:14.693054 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:14.693022 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:20:14.693950 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:14.693931 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:20:15.031771 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:15.031734 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27d5b50e-dnxq8"] Oct 30 07:20:15.034671 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:15.034641 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27d5b50e-dnxq8"] Oct 30 07:20:15.773593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:15.773544 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc5c70db-419b-439d-aa07-a9bdfc78aad0" path="/var/lib/kubelet/pods/fc5c70db-419b-439d-aa07-a9bdfc78aad0/volumes" Oct 30 07:20:16.356550 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:16.356515 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k"] Oct 30 07:20:16.360697 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:16.360668 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k" Oct 30 07:20:16.390283 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:16.390244 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k"] Oct 30 07:20:16.390493 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:16.390475 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-ab3d83-serving-cert\"" Oct 30 07:20:16.441608 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:16.441559 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0e72c9-f992-4f9d-9da3-11ca496ac9e2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k\" (UID: \"ce0e72c9-f992-4f9d-9da3-11ca496ac9e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k" Oct 30 07:20:16.441806 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:16.441701 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0e72c9-f992-4f9d-9da3-11ca496ac9e2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k\" (UID: \"ce0e72c9-f992-4f9d-9da3-11ca496ac9e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k" Oct 30 07:20:16.441891 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:16.441872 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0e72c9-f992-4f9d-9da3-11ca496ac9e2-metrics-certs\") pod \"maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k\" (UID: \"ce0e72c9-f992-4f9d-9da3-11ca496ac9e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k" Oct 30 07:20:16.442026 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:16.442005 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gps9f\" (UniqueName: \"kubernetes.io/projected/ce0e72c9-f992-4f9d-9da3-11ca496ac9e2-kube-api-access-gps9f\") pod \"maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k\" (UID: \"ce0e72c9-f992-4f9d-9da3-11ca496ac9e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k" Oct 30 07:20:16.583594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:16.562559 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gps9f\" (UniqueName: \"kubernetes.io/projected/ce0e72c9-f992-4f9d-9da3-11ca496ac9e2-kube-api-access-gps9f\") pod \"maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k\" (UID: \"ce0e72c9-f992-4f9d-9da3-11ca496ac9e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k" Oct 30 07:20:16.583594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:16.562721 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0e72c9-f992-4f9d-9da3-11ca496ac9e2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k\" (UID: \"ce0e72c9-f992-4f9d-9da3-11ca496ac9e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k" Oct 30 07:20:16.583594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:16.562775 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0e72c9-f992-4f9d-9da3-11ca496ac9e2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k\" (UID: \"ce0e72c9-f992-4f9d-9da3-11ca496ac9e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k" Oct 30 07:20:16.583594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:16.562911 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0e72c9-f992-4f9d-9da3-11ca496ac9e2-metrics-certs\") pod \"maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k\" (UID: \"ce0e72c9-f992-4f9d-9da3-11ca496ac9e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k" Oct 30 07:20:16.583594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:16.578141 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0e72c9-f992-4f9d-9da3-11ca496ac9e2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k\" (UID: \"ce0e72c9-f992-4f9d-9da3-11ca496ac9e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k" Oct 30 07:20:16.617685 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:16.606395 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0e72c9-f992-4f9d-9da3-11ca496ac9e2-metrics-certs\") pod \"maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k\" (UID: \"ce0e72c9-f992-4f9d-9da3-11ca496ac9e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k" Oct 30 07:20:16.623794 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:16.622068 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0e72c9-f992-4f9d-9da3-11ca496ac9e2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k\" (UID: \"ce0e72c9-f992-4f9d-9da3-11ca496ac9e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k" Oct 30 07:20:16.648598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:16.635373 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gps9f\" (UniqueName: \"kubernetes.io/projected/ce0e72c9-f992-4f9d-9da3-11ca496ac9e2-kube-api-access-gps9f\") pod \"maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k\" (UID: \"ce0e72c9-f992-4f9d-9da3-11ca496ac9e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k" Oct 30 07:20:16.699600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:16.689617 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k" Oct 30 07:20:17.080284 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:17.080252 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k"] Oct 30 07:20:17.081320 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:20:17.081298 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0e72c9_f992_4f9d_9da3_11ca496ac9e2.slice/crio-4e7158ea1e42fd8ebc5f413044e54101b27fd738f48692b7649c1cddfc63a55b WatchSource:0}: Error finding container 4e7158ea1e42fd8ebc5f413044e54101b27fd738f48692b7649c1cddfc63a55b: Status 404 returned error can't find the container with id 4e7158ea1e42fd8ebc5f413044e54101b27fd738f48692b7649c1cddfc63a55b Oct 30 07:20:17.082936 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:17.082916 4951 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 07:20:17.911547 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:17.911503 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k" event={"ID":"ce0e72c9-f992-4f9d-9da3-11ca496ac9e2","Type":"ContainerStarted","Data":"9194bcbd9b62bc5b9a3952318bbdff322aef4415c38f6ae461ee9f490f8c1e26"} Oct 30 07:20:17.917270 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:17.914043 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k" podStartSLOduration=1.536361156 podStartE2EDuration="1.91402109s" totalImagesPullingTime="377.659934ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:20:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:20:17.909979777 +0000 UTC m=+1424.253027744" watchObservedRunningTime="2026-10-30 07:20:17.91402109 +0000 UTC m=+1424.257069058" Oct 30 07:20:17.940470 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:17.940256 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k" Oct 30 07:20:17.940470 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:17.940336 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k" event={"ID":"ce0e72c9-f992-4f9d-9da3-11ca496ac9e2","Type":"ContainerStarted","Data":"4e7158ea1e42fd8ebc5f413044e54101b27fd738f48692b7649c1cddfc63a55b"} Oct 30 07:20:18.198321 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:18.198240 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-655945b85f-qhdlr"] Oct 30 07:20:18.201759 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:18.201737 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-655945b85f-qhdlr" Oct 30 07:20:18.222083 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:18.222056 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3d2270\"" Oct 30 07:20:18.222280 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:18.222255 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3d2270-dockercfg-fnv9c\"" Oct 30 07:20:18.222873 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:18.222853 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-655945b85f-qhdlr"] Oct 30 07:20:18.343801 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:18.343709 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3d2270-5db8699bbb-56xv9"] Oct 30 07:20:18.493467 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:18.445622 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8889754c-55d5-4722-94dc-9579494fdd3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3d2270-655945b85f-qhdlr\" (UID: \"8889754c-55d5-4722-94dc-9579494fdd3e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-655945b85f-qhdlr" Oct 30 07:20:18.493467 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:18.445750 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jgkr\" (UniqueName: \"kubernetes.io/projected/8889754c-55d5-4722-94dc-9579494fdd3e-kube-api-access-9jgkr\") pod \"payload-pre-processing-e2e-ipp-b-3d2270-655945b85f-qhdlr\" (UID: \"8889754c-55d5-4722-94dc-9579494fdd3e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-655945b85f-qhdlr" Oct 30 07:20:18.497181 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:18.494864 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-5db8699bbb-56xv9" Oct 30 07:20:18.499249 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:18.499210 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3d2270-5db8699bbb-56xv9"] Oct 30 07:20:18.547741 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:18.547704 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jgkr\" (UniqueName: \"kubernetes.io/projected/8889754c-55d5-4722-94dc-9579494fdd3e-kube-api-access-9jgkr\") pod \"payload-pre-processing-e2e-ipp-b-3d2270-655945b85f-qhdlr\" (UID: \"8889754c-55d5-4722-94dc-9579494fdd3e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-655945b85f-qhdlr" Oct 30 07:20:18.547919 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:18.547752 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3d2270-5db8699bbb-56xv9\" (UID: \"91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-5db8699bbb-56xv9" Oct 30 07:20:18.547973 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:18.547950 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8889754c-55d5-4722-94dc-9579494fdd3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3d2270-655945b85f-qhdlr\" (UID: \"8889754c-55d5-4722-94dc-9579494fdd3e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-655945b85f-qhdlr" Oct 30 07:20:18.548031 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:18.547980 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpxmm\" (UniqueName: \"kubernetes.io/projected/91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e-kube-api-access-vpxmm\") pod \"payload-processing-e2e-ipp-b-3d2270-5db8699bbb-56xv9\" (UID: \"91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-5db8699bbb-56xv9" Oct 30 07:20:18.548737 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:18.548710 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8889754c-55d5-4722-94dc-9579494fdd3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3d2270-655945b85f-qhdlr\" (UID: \"8889754c-55d5-4722-94dc-9579494fdd3e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-655945b85f-qhdlr" Oct 30 07:20:18.570675 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:18.570646 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jgkr\" (UniqueName: \"kubernetes.io/projected/8889754c-55d5-4722-94dc-9579494fdd3e-kube-api-access-9jgkr\") pod \"payload-pre-processing-e2e-ipp-b-3d2270-655945b85f-qhdlr\" (UID: \"8889754c-55d5-4722-94dc-9579494fdd3e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-655945b85f-qhdlr" Oct 30 07:20:18.649431 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:18.649389 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpxmm\" (UniqueName: \"kubernetes.io/projected/91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e-kube-api-access-vpxmm\") pod \"payload-processing-e2e-ipp-b-3d2270-5db8699bbb-56xv9\" (UID: \"91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-5db8699bbb-56xv9" Oct 30 07:20:18.649642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:18.649474 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3d2270-5db8699bbb-56xv9\" (UID: \"91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-5db8699bbb-56xv9" Oct 30 07:20:18.650433 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:18.650406 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3d2270-5db8699bbb-56xv9\" (UID: \"91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-5db8699bbb-56xv9" Oct 30 07:20:18.673439 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:18.673398 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpxmm\" (UniqueName: \"kubernetes.io/projected/91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e-kube-api-access-vpxmm\") pod \"payload-processing-e2e-ipp-b-3d2270-5db8699bbb-56xv9\" (UID: \"91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-5db8699bbb-56xv9" Oct 30 07:20:18.816357 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:18.816313 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-655945b85f-qhdlr" Oct 30 07:20:18.817079 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:18.816938 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-5db8699bbb-56xv9" Oct 30 07:20:19.214978 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:20:19.214936 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8889754c_55d5_4722_94dc_9579494fdd3e.slice/crio-ad26a6efbfa5d60cb24d2ba8e22c23a4cdfab84854c3c9ab5de7dcd02f38af1e WatchSource:0}: Error finding container ad26a6efbfa5d60cb24d2ba8e22c23a4cdfab84854c3c9ab5de7dcd02f38af1e: Status 404 returned error can't find the container with id ad26a6efbfa5d60cb24d2ba8e22c23a4cdfab84854c3c9ab5de7dcd02f38af1e Oct 30 07:20:19.217187 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.217155 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-655945b85f-qhdlr"] Oct 30 07:20:19.238039 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.238014 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3d2270-5db8699bbb-56xv9"] Oct 30 07:20:19.238706 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:20:19.238679 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91a4673f_4bc9_4d03_8fe2_72d8b9a51a4e.slice/crio-68172bfd1708401c6b22568630e31a5e0d805cdb260fa0535ad6254e0ce64bbf WatchSource:0}: Error finding container 68172bfd1708401c6b22568630e31a5e0d805cdb260fa0535ad6254e0ce64bbf: Status 404 returned error can't find the container with id 68172bfd1708401c6b22568630e31a5e0d805cdb260fa0535ad6254e0ce64bbf Oct 30 07:20:19.651663 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.651628 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn"] Oct 30 07:20:19.656173 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.656149 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:20:19.672761 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.672739 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-efce7b-openshift-default-dockercfg-54g6m\"" Oct 30 07:20:19.680146 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.680123 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvctj\" (UniqueName: \"kubernetes.io/projected/5c1b682b-01eb-4717-9184-d81851ade4b0-kube-api-access-nvctj\") pod \"e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:20:19.680284 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.680158 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c1b682b-01eb-4717-9184-d81851ade4b0-istiod-ca-cert\") pod \"e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:20:19.680284 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.680177 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-istio-envoy\") pod \"e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:20:19.680400 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.680301 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-workload-certs\") pod \"e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:20:19.680400 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.680364 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c1b682b-01eb-4717-9184-d81851ade4b0-istio-token\") pod \"e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:20:19.680508 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.680487 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-credential-socket\") pod \"e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:20:19.680565 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.680540 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-workload-socket\") pod \"e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:20:19.680636 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.680617 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-istio-data\") pod \"e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:20:19.680699 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.680685 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c1b682b-01eb-4717-9184-d81851ade4b0-istio-podinfo\") pod \"e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:20:19.706284 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.706237 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn"] Oct 30 07:20:19.877826 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.861898 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvctj\" (UniqueName: \"kubernetes.io/projected/5c1b682b-01eb-4717-9184-d81851ade4b0-kube-api-access-nvctj\") pod \"e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:20:19.894642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.889719 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c1b682b-01eb-4717-9184-d81851ade4b0-istiod-ca-cert\") pod \"e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:20:19.907108 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.890586 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c1b682b-01eb-4717-9184-d81851ade4b0-istiod-ca-cert\") pod \"e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:20:19.907108 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.906029 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-istio-envoy\") pod \"e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:20:19.907108 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.906099 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-workload-certs\") pod \"e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:20:19.907108 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.906168 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c1b682b-01eb-4717-9184-d81851ade4b0-istio-token\") pod \"e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:20:19.907108 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.906242 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-credential-socket\") pod \"e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:20:19.907108 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.906306 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-workload-socket\") pod \"e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:20:19.907108 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.906342 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-istio-data\") pod \"e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:20:19.907108 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.906461 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c1b682b-01eb-4717-9184-d81851ade4b0-istio-podinfo\") pod \"e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:20:19.907108 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.906993 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-workload-certs\") pod \"e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:20:19.911601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.910081 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-istio-envoy\") pod \"e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:20:19.928475 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.928429 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-credential-socket\") pod \"e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:20:19.928785 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.928761 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-istio-data\") pod \"e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:20:19.928972 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.928952 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-workload-socket\") pod \"e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:20:19.933471 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.933422 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvctj\" (UniqueName: \"kubernetes.io/projected/5c1b682b-01eb-4717-9184-d81851ade4b0-kube-api-access-nvctj\") pod \"e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:20:19.936337 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.936209 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c1b682b-01eb-4717-9184-d81851ade4b0-istio-podinfo\") pod \"e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:20:19.937005 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.936978 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c1b682b-01eb-4717-9184-d81851ade4b0-istio-token\") pod \"e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:20:19.977984 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.977941 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-5db8699bbb-56xv9" event={"ID":"91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e","Type":"ContainerStarted","Data":"3a1dd9ad5b4845566d5e0e829f030f7281696af57881ddc51cfe7fac0b26a923"} Oct 30 07:20:19.978234 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.978217 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-5db8699bbb-56xv9" event={"ID":"91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e","Type":"ContainerStarted","Data":"68172bfd1708401c6b22568630e31a5e0d805cdb260fa0535ad6254e0ce64bbf"} Oct 30 07:20:19.979893 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:19.979867 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-5db8699bbb-56xv9" Oct 30 07:20:20.007009 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:20.006977 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:20:20.011388 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:20.011348 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-655945b85f-qhdlr" event={"ID":"8889754c-55d5-4722-94dc-9579494fdd3e","Type":"ContainerStarted","Data":"70b63a6cb11f9ccb19c5b25baf2cdb3d18621666ba8ba5f88b84d42c3fa4e505"} Oct 30 07:20:20.011557 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:20.011418 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-655945b85f-qhdlr" event={"ID":"8889754c-55d5-4722-94dc-9579494fdd3e","Type":"ContainerStarted","Data":"ad26a6efbfa5d60cb24d2ba8e22c23a4cdfab84854c3c9ab5de7dcd02f38af1e"} Oct 30 07:20:20.012043 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:20.012023 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-655945b85f-qhdlr" Oct 30 07:20:20.030323 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:20.030264 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-5db8699bbb-56xv9" podStartSLOduration=2.030247484 podStartE2EDuration="2.030247484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:20:20.023946953 +0000 UTC m=+1426.366994919" watchObservedRunningTime="2026-10-30 07:20:20.030247484 +0000 UTC m=+1426.373295451" Oct 30 07:20:20.066283 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:20.066235 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-655945b85f-qhdlr" podStartSLOduration=2.066220596 podStartE2EDuration="2.066220596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:20:20.063233819 +0000 UTC m=+1426.406281784" watchObservedRunningTime="2026-10-30 07:20:20.066220596 +0000 UTC m=+1426.409268560" Oct 30 07:20:20.348248 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:20.348210 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn"] Oct 30 07:20:20.348706 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:20:20.348392 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c1b682b_01eb_4717_9184_d81851ade4b0.slice/crio-7227635c5a8ae197117eb6c0367e827adfec48627a82872c2fcc252e0f7c3c29 WatchSource:0}: Error finding container 7227635c5a8ae197117eb6c0367e827adfec48627a82872c2fcc252e0f7c3c29: Status 404 returned error can't find the container with id 7227635c5a8ae197117eb6c0367e827adfec48627a82872c2fcc252e0f7c3c29 Oct 30 07:20:20.351297 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:20.351258 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:20:20.351408 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:20.351373 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:20:20.351449 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:20.351422 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:20:21.021195 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:21.021154 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" event={"ID":"5c1b682b-01eb-4717-9184-d81851ade4b0","Type":"ContainerStarted","Data":"fa2c7640c0bc198242c773e7da1f563fb2c1ccd7fbeec578cf012036992e4450"} Oct 30 07:20:21.021410 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:21.021223 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" event={"ID":"5c1b682b-01eb-4717-9184-d81851ade4b0","Type":"ContainerStarted","Data":"7227635c5a8ae197117eb6c0367e827adfec48627a82872c2fcc252e0f7c3c29"} Oct 30 07:20:21.086472 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:21.086404 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" podStartSLOduration=2.086383952 podStartE2EDuration="2.086383952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:20:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:20:21.077322955 +0000 UTC m=+1427.420370942" watchObservedRunningTime="2026-10-30 07:20:21.086383952 +0000 UTC m=+1427.429431919" Oct 30 07:20:22.007931 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:22.007903 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:20:22.013513 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:22.013480 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:20:22.029899 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:22.029861 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:20:22.030771 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:22.030741 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:20:23.062628 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:20:23.062594 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:20:23.228896 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:23.228850 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-052254-589746f98c-l74cl"] Oct 30 07:20:23.877531 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:23.877504 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k" Oct 30 07:20:25.407842 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:25.407809 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-68bf65d8cf-lgp4z"] Oct 30 07:20:25.412448 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:25.412419 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-68bf65d8cf-lgp4z" Oct 30 07:20:25.441649 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:25.441618 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-68bf65d8cf-lgp4z"] Oct 30 07:20:25.499532 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:25.499499 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-efce7b\"" Oct 30 07:20:25.510605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:25.499970 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-efce7b-dockercfg-rzgwg\"" Oct 30 07:20:25.632654 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:25.631359 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e4e9141-878d-424a-a658-87daf4272ddf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-efce7b-68bf65d8cf-lgp4z\" (UID: \"5e4e9141-878d-424a-a658-87daf4272ddf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-68bf65d8cf-lgp4z" Oct 30 07:20:25.632654 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:25.631470 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8x8w\" (UniqueName: \"kubernetes.io/projected/5e4e9141-878d-424a-a658-87daf4272ddf-kube-api-access-z8x8w\") pod \"payload-pre-processing-e2e-ipp-b-efce7b-68bf65d8cf-lgp4z\" (UID: \"5e4e9141-878d-424a-a658-87daf4272ddf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-68bf65d8cf-lgp4z" Oct 30 07:20:25.741210 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:25.735732 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e4e9141-878d-424a-a658-87daf4272ddf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-efce7b-68bf65d8cf-lgp4z\" (UID: \"5e4e9141-878d-424a-a658-87daf4272ddf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-68bf65d8cf-lgp4z" Oct 30 07:20:25.741210 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:25.735834 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8x8w\" (UniqueName: \"kubernetes.io/projected/5e4e9141-878d-424a-a658-87daf4272ddf-kube-api-access-z8x8w\") pod \"payload-pre-processing-e2e-ipp-b-efce7b-68bf65d8cf-lgp4z\" (UID: \"5e4e9141-878d-424a-a658-87daf4272ddf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-68bf65d8cf-lgp4z" Oct 30 07:20:25.741210 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:25.737303 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e4e9141-878d-424a-a658-87daf4272ddf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-efce7b-68bf65d8cf-lgp4z\" (UID: \"5e4e9141-878d-424a-a658-87daf4272ddf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-68bf65d8cf-lgp4z" Oct 30 07:20:25.759716 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:25.759683 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8x8w\" (UniqueName: \"kubernetes.io/projected/5e4e9141-878d-424a-a658-87daf4272ddf-kube-api-access-z8x8w\") pod \"payload-pre-processing-e2e-ipp-b-efce7b-68bf65d8cf-lgp4z\" (UID: \"5e4e9141-878d-424a-a658-87daf4272ddf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-68bf65d8cf-lgp4z" Oct 30 07:20:26.030894 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:26.030859 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-68bf65d8cf-lgp4z" Oct 30 07:20:26.361415 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:20:26.361364 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e4e9141_878d_424a_a658_87daf4272ddf.slice/crio-0091d406a4d3762097d8d08fc085b4cc07436dd45e5790b8b36b97adb1f8e9e6 WatchSource:0}: Error finding container 0091d406a4d3762097d8d08fc085b4cc07436dd45e5790b8b36b97adb1f8e9e6: Status 404 returned error can't find the container with id 0091d406a4d3762097d8d08fc085b4cc07436dd45e5790b8b36b97adb1f8e9e6 Oct 30 07:20:26.363099 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:26.363069 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-68bf65d8cf-lgp4z"] Oct 30 07:20:27.150057 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:27.150015 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-68bf65d8cf-lgp4z" event={"ID":"5e4e9141-878d-424a-a658-87daf4272ddf","Type":"ContainerStarted","Data":"b423a31dfcf62a0367e1bf8b845219f6c60551a376e21cf9d8d4de5e80294222"} Oct 30 07:20:27.150516 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:27.150087 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-68bf65d8cf-lgp4z" event={"ID":"5e4e9141-878d-424a-a658-87daf4272ddf","Type":"ContainerStarted","Data":"0091d406a4d3762097d8d08fc085b4cc07436dd45e5790b8b36b97adb1f8e9e6"} Oct 30 07:20:27.150516 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:27.150153 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-68bf65d8cf-lgp4z" Oct 30 07:20:27.194540 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:27.194492 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-68bf65d8cf-lgp4z" podStartSLOduration=2.194478604 podStartE2EDuration="2.194478604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:20:27.190325067 +0000 UTC m=+1433.533373032" watchObservedRunningTime="2026-10-30 07:20:27.194478604 +0000 UTC m=+1433.537526603" Oct 30 07:20:28.229987 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:28.229945 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-052254-589746f98c-l74cl" podUID="96900a43-4be1-46f0-ab3f-9162fa89f534" containerName="payload-pre-processing" containerID="cri-o://9949ebc6fde83a21bea18d39a7901cb960a4e5d143cb6ac212e592372aa73b7e" gracePeriod=25 Oct 30 07:20:28.624277 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:28.624245 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-052254-589746f98c-l74cl" Oct 30 07:20:28.792635 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:28.792540 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96900a43-4be1-46f0-ab3f-9162fa89f534-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96900a43-4be1-46f0-ab3f-9162fa89f534-plugins-config-volume\") pod \"96900a43-4be1-46f0-ab3f-9162fa89f534\" (UID: \"96900a43-4be1-46f0-ab3f-9162fa89f534\") " Oct 30 07:20:28.792798 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:28.792667 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96900a43-4be1-46f0-ab3f-9162fa89f534-kube-api-access-79ld5\" (UniqueName: \"kubernetes.io/projected/96900a43-4be1-46f0-ab3f-9162fa89f534-kube-api-access-79ld5\") pod \"96900a43-4be1-46f0-ab3f-9162fa89f534\" (UID: \"96900a43-4be1-46f0-ab3f-9162fa89f534\") " Oct 30 07:20:28.793025 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:28.792998 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96900a43-4be1-46f0-ab3f-9162fa89f534-plugins-config-volume" pod "96900a43-4be1-46f0-ab3f-9162fa89f534" (UID: "96900a43-4be1-46f0-ab3f-9162fa89f534"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:20:28.793136 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:28.793122 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96900a43-4be1-46f0-ab3f-9162fa89f534-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:28.794878 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:28.794856 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96900a43-4be1-46f0-ab3f-9162fa89f534-kube-api-access-79ld5" pod "96900a43-4be1-46f0-ab3f-9162fa89f534" (UID: "96900a43-4be1-46f0-ab3f-9162fa89f534"). InnerVolumeSpecName "kube-api-access-79ld5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:20:28.894330 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:28.894289 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79ld5\" (UniqueName: \"kubernetes.io/projected/96900a43-4be1-46f0-ab3f-9162fa89f534-kube-api-access-79ld5\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:29.169425 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:29.169325 4951 generic.go:356] "Generic (PLEG): container finished" podID="96900a43-4be1-46f0-ab3f-9162fa89f534" containerID="9949ebc6fde83a21bea18d39a7901cb960a4e5d143cb6ac212e592372aa73b7e" exitCode=0 Oct 30 07:20:29.169425 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:29.169407 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-052254-589746f98c-l74cl" Oct 30 07:20:29.169703 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:29.169409 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-052254-589746f98c-l74cl" event={"ID":"96900a43-4be1-46f0-ab3f-9162fa89f534","Type":"ContainerDied","Data":"9949ebc6fde83a21bea18d39a7901cb960a4e5d143cb6ac212e592372aa73b7e"} Oct 30 07:20:29.169703 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:29.169564 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-052254-589746f98c-l74cl" event={"ID":"96900a43-4be1-46f0-ab3f-9162fa89f534","Type":"ContainerDied","Data":"173add32c5d574f33dfa80c45623395e97297f5bb38395dfcc9bd59801036cec"} Oct 30 07:20:29.169703 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:29.169622 4951 scope.go:122] "RemoveContainer" containerID="9949ebc6fde83a21bea18d39a7901cb960a4e5d143cb6ac212e592372aa73b7e" Oct 30 07:20:29.180548 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:29.180527 4951 scope.go:122] "RemoveContainer" containerID="9949ebc6fde83a21bea18d39a7901cb960a4e5d143cb6ac212e592372aa73b7e" Oct 30 07:20:29.180867 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:20:29.180846 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9949ebc6fde83a21bea18d39a7901cb960a4e5d143cb6ac212e592372aa73b7e\": container with ID starting with 9949ebc6fde83a21bea18d39a7901cb960a4e5d143cb6ac212e592372aa73b7e not found: ID does not exist" containerID="9949ebc6fde83a21bea18d39a7901cb960a4e5d143cb6ac212e592372aa73b7e" Oct 30 07:20:29.180910 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:29.180877 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9949ebc6fde83a21bea18d39a7901cb960a4e5d143cb6ac212e592372aa73b7e"} err="failed to get container status \"9949ebc6fde83a21bea18d39a7901cb960a4e5d143cb6ac212e592372aa73b7e\": rpc error: code = NotFound desc = could not find container \"9949ebc6fde83a21bea18d39a7901cb960a4e5d143cb6ac212e592372aa73b7e\": container with ID starting with 9949ebc6fde83a21bea18d39a7901cb960a4e5d143cb6ac212e592372aa73b7e not found: ID does not exist" Oct 30 07:20:29.221250 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:29.221215 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-052254-589746f98c-l74cl"] Oct 30 07:20:29.233332 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:29.233306 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-052254-589746f98c-l74cl"] Oct 30 07:20:29.651680 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:29.650013 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-486cd4-79d7766689-przsr"] Oct 30 07:20:29.651680 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:29.650332 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-486cd4-79d7766689-przsr" podUID="e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4" containerName="maas-api" containerID="cri-o://b08ea5bc595cbaae109f1a370dff033347034e547ee333e1f28ef3248564ba13" gracePeriod=30 Oct 30 07:20:29.773430 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:29.773399 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96900a43-4be1-46f0-ab3f-9162fa89f534" path="/var/lib/kubelet/pods/96900a43-4be1-46f0-ab3f-9162fa89f534/volumes" Oct 30 07:20:30.273896 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:30.273862 4951 generic.go:356] "Generic (PLEG): container finished" podID="e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4" containerID="b08ea5bc595cbaae109f1a370dff033347034e547ee333e1f28ef3248564ba13" exitCode=0 Oct 30 07:20:30.274805 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:30.274779 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-486cd4-79d7766689-przsr" event={"ID":"e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4","Type":"ContainerDied","Data":"b08ea5bc595cbaae109f1a370dff033347034e547ee333e1f28ef3248564ba13"} Oct 30 07:20:30.279375 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:30.279351 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-486cd4-79d7766689-przsr" Oct 30 07:20:30.478987 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:30.478890 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4-openshift-service-ca\") pod \"e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4\" (UID: \"e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4\") " Oct 30 07:20:30.479262 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:30.479221 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4-maas-api-tls\") pod \"e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4\" (UID: \"e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4\") " Oct 30 07:20:30.479538 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:30.479513 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4-kube-api-access-nzpk7\" (UniqueName: \"kubernetes.io/projected/e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4-kube-api-access-nzpk7\") pod \"e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4\" (UID: \"e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4\") " Oct 30 07:20:30.479898 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:30.479868 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4-metrics-certs\") pod \"e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4\" (UID: \"e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4\") " Oct 30 07:20:30.500479 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:30.500422 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4-openshift-service-ca" pod "e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4" (UID: "e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:20:30.515533 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:30.515363 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4-kube-api-access-nzpk7" pod "e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4" (UID: "e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4"). InnerVolumeSpecName "kube-api-access-nzpk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:20:30.516080 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:30.516051 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4-metrics-certs" pod "e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4" (UID: "e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:20:30.529354 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:30.529312 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4-maas-api-tls" pod "e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4" (UID: "e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:20:30.583521 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:30.583460 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:30.583521 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:30.583516 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:30.583521 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:30.583527 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzpk7\" (UniqueName: \"kubernetes.io/projected/e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4-kube-api-access-nzpk7\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:30.583791 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:30.583538 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:31.024143 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:31.024112 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-655945b85f-qhdlr" Oct 30 07:20:31.282556 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:31.282472 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-486cd4-79d7766689-przsr" Oct 30 07:20:31.282556 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:31.282498 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-486cd4-79d7766689-przsr" event={"ID":"e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4","Type":"ContainerDied","Data":"31849c59d900615d8f314457bd7f37f98dbf42bebe0680401d3c1dabe784c31f"} Oct 30 07:20:31.283082 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:31.282568 4951 scope.go:122] "RemoveContainer" containerID="b08ea5bc595cbaae109f1a370dff033347034e547ee333e1f28ef3248564ba13" Oct 30 07:20:31.337712 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:31.337679 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-486cd4-79d7766689-przsr"] Oct 30 07:20:31.349545 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:31.349519 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-486cd4-79d7766689-przsr"] Oct 30 07:20:31.774194 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:31.774159 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4" path="/var/lib/kubelet/pods/e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4/volumes" Oct 30 07:20:32.044475 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:32.044383 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-5db8699bbb-56xv9" Oct 30 07:20:34.898534 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:34.898445 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn"] Oct 30 07:20:34.899335 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:34.899310 4951 cpu_manager.go:414] "containerMap: removing container" podUID="96900a43-4be1-46f0-ab3f-9162fa89f534" containerName="payload-pre-processing" Oct 30 07:20:34.899446 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:34.899335 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96900a43-4be1-46f0-ab3f-9162fa89f534" containerName="payload-pre-processing" Oct 30 07:20:34.899446 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:34.899398 4951 cpu_manager.go:414] "containerMap: removing container" podUID="e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4" containerName="maas-api" Oct 30 07:20:34.899446 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:34.899407 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4" containerName="maas-api" Oct 30 07:20:34.899727 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:34.899559 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="e67ce57d-fbc0-49f6-b16c-fa7a5256a6e4" containerName="maas-api" Oct 30 07:20:34.899727 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:34.899598 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="96900a43-4be1-46f0-ab3f-9162fa89f534" containerName="payload-pre-processing" Oct 30 07:20:34.906240 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:34.906219 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn" Oct 30 07:20:34.937814 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:34.937786 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-3d2270-serving-cert\"" Oct 30 07:20:34.979716 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:34.968641 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn"] Oct 30 07:20:35.079097 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:35.078847 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b02dc8b-0505-4f42-ad86-4301ddfbaf6e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn\" (UID: \"2b02dc8b-0505-4f42-ad86-4301ddfbaf6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn" Oct 30 07:20:35.079097 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:35.078943 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b02dc8b-0505-4f42-ad86-4301ddfbaf6e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn\" (UID: \"2b02dc8b-0505-4f42-ad86-4301ddfbaf6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn" Oct 30 07:20:35.079356 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:35.079121 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntsf8\" (UniqueName: \"kubernetes.io/projected/2b02dc8b-0505-4f42-ad86-4301ddfbaf6e-kube-api-access-ntsf8\") pod \"maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn\" (UID: \"2b02dc8b-0505-4f42-ad86-4301ddfbaf6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn" Oct 30 07:20:35.079356 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:35.079189 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b02dc8b-0505-4f42-ad86-4301ddfbaf6e-metrics-certs\") pod \"maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn\" (UID: \"2b02dc8b-0505-4f42-ad86-4301ddfbaf6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn" Oct 30 07:20:35.186799 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:35.183960 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntsf8\" (UniqueName: \"kubernetes.io/projected/2b02dc8b-0505-4f42-ad86-4301ddfbaf6e-kube-api-access-ntsf8\") pod \"maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn\" (UID: \"2b02dc8b-0505-4f42-ad86-4301ddfbaf6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn" Oct 30 07:20:35.186799 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:35.184067 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b02dc8b-0505-4f42-ad86-4301ddfbaf6e-metrics-certs\") pod \"maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn\" (UID: \"2b02dc8b-0505-4f42-ad86-4301ddfbaf6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn" Oct 30 07:20:35.186799 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:35.184203 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b02dc8b-0505-4f42-ad86-4301ddfbaf6e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn\" (UID: \"2b02dc8b-0505-4f42-ad86-4301ddfbaf6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn" Oct 30 07:20:35.186799 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:35.184271 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b02dc8b-0505-4f42-ad86-4301ddfbaf6e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn\" (UID: \"2b02dc8b-0505-4f42-ad86-4301ddfbaf6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn" Oct 30 07:20:35.186799 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:35.185213 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b02dc8b-0505-4f42-ad86-4301ddfbaf6e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn\" (UID: \"2b02dc8b-0505-4f42-ad86-4301ddfbaf6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn" Oct 30 07:20:35.212593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:35.212435 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntsf8\" (UniqueName: \"kubernetes.io/projected/2b02dc8b-0505-4f42-ad86-4301ddfbaf6e-kube-api-access-ntsf8\") pod \"maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn\" (UID: \"2b02dc8b-0505-4f42-ad86-4301ddfbaf6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn" Oct 30 07:20:35.216736 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:35.216482 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b02dc8b-0505-4f42-ad86-4301ddfbaf6e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn\" (UID: \"2b02dc8b-0505-4f42-ad86-4301ddfbaf6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn" Oct 30 07:20:35.216736 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:35.216689 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b02dc8b-0505-4f42-ad86-4301ddfbaf6e-metrics-certs\") pod \"maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn\" (UID: \"2b02dc8b-0505-4f42-ad86-4301ddfbaf6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn" Oct 30 07:20:35.220210 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:35.220186 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn" Oct 30 07:20:35.565557 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:35.565379 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn"] Oct 30 07:20:35.565557 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:20:35.565430 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b02dc8b_0505_4f42_ad86_4301ddfbaf6e.slice/crio-397cf6f75ecc1ff2f9e5745c7b191076c7e857794577bff3525c4a8c15a11141 WatchSource:0}: Error finding container 397cf6f75ecc1ff2f9e5745c7b191076c7e857794577bff3525c4a8c15a11141: Status 404 returned error can't find the container with id 397cf6f75ecc1ff2f9e5745c7b191076c7e857794577bff3525c4a8c15a11141 Oct 30 07:20:36.381355 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:36.381311 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn" event={"ID":"2b02dc8b-0505-4f42-ad86-4301ddfbaf6e","Type":"ContainerStarted","Data":"b5d85a022dda9a624e686db2dc6d6b56e26c5caced738fd1f0b07d25d051b005"} Oct 30 07:20:36.381803 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:36.381380 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn" Oct 30 07:20:36.381803 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:36.381401 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn" event={"ID":"2b02dc8b-0505-4f42-ad86-4301ddfbaf6e","Type":"ContainerStarted","Data":"397cf6f75ecc1ff2f9e5745c7b191076c7e857794577bff3525c4a8c15a11141"} Oct 30 07:20:36.463197 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:36.462063 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn" podStartSLOduration=2.072740575 podStartE2EDuration="2.462042206s" totalImagesPullingTime="389.301631ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:20:36.442126958 +0000 UTC m=+1442.785174947" watchObservedRunningTime="2026-10-30 07:20:36.462042206 +0000 UTC m=+1442.805090171" Oct 30 07:20:38.164523 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:38.164487 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-68bf65d8cf-lgp4z" Oct 30 07:20:38.259378 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:38.259322 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v"] Oct 30 07:20:38.259629 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:38.259598 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" podUID="f3d3f735-6cd0-4153-a6ff-88f77f4941ca" containerName="istio-proxy" containerID="cri-o://e7c7a4790b3a792a5a717ead1f3d71eaa2577132870247c020633cd6e3c619a2" gracePeriod=30 Oct 30 07:20:39.850531 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:39.844707 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv"] Oct 30 07:20:39.881274 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:39.876849 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:20:39.887293 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:39.887246 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv"] Oct 30 07:20:39.894495 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:39.894449 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-17bd11-openshift-default-dockercfg-gdkpt\"" Oct 30 07:20:39.978284 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:39.978246 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-workload-socket\") pod \"e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:20:39.978446 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:39.978312 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0938f71-f1d8-4342-8d0a-7aed59adef39-istiod-ca-cert\") pod \"e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:20:39.978497 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:39.978484 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-istio-data\") pod \"e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:20:39.978640 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:39.978613 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88s8w\" (UniqueName: \"kubernetes.io/projected/b0938f71-f1d8-4342-8d0a-7aed59adef39-kube-api-access-88s8w\") pod \"e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:20:39.978724 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:39.978710 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0938f71-f1d8-4342-8d0a-7aed59adef39-istio-token\") pod \"e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:20:39.978775 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:39.978752 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-credential-socket\") pod \"e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:20:39.978829 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:39.978816 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0938f71-f1d8-4342-8d0a-7aed59adef39-istio-podinfo\") pod \"e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:20:39.978883 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:39.978855 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-workload-certs\") pod \"e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:20:39.978969 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:39.978942 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-istio-envoy\") pod \"e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:20:40.116112 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:40.084713 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88s8w\" (UniqueName: \"kubernetes.io/projected/b0938f71-f1d8-4342-8d0a-7aed59adef39-kube-api-access-88s8w\") pod \"e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:20:40.116112 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:40.084846 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0938f71-f1d8-4342-8d0a-7aed59adef39-istio-token\") pod \"e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:20:40.116112 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:40.084892 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-credential-socket\") pod \"e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:20:40.116112 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:40.084949 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0938f71-f1d8-4342-8d0a-7aed59adef39-istio-podinfo\") pod \"e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:20:40.116112 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:40.084986 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-workload-certs\") pod \"e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:20:40.116112 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:40.085074 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-istio-envoy\") pod \"e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:20:40.116112 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:40.085164 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-workload-socket\") pod \"e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:20:40.116112 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:40.085208 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0938f71-f1d8-4342-8d0a-7aed59adef39-istiod-ca-cert\") pod \"e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:20:40.116112 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:40.085287 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-istio-data\") pod \"e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:20:40.116112 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:40.085802 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-istio-data\") pod \"e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:20:40.116112 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:40.086595 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-credential-socket\") pod \"e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:20:40.116112 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:40.096361 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-workload-certs\") pod \"e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:20:40.116112 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:40.096637 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-workload-socket\") pod \"e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:20:40.116112 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:40.097254 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0938f71-f1d8-4342-8d0a-7aed59adef39-istiod-ca-cert\") pod \"e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:20:40.116112 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:40.098522 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0938f71-f1d8-4342-8d0a-7aed59adef39-istio-podinfo\") pod \"e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:20:40.116112 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:40.115995 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-istio-envoy\") pod \"e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:20:40.131299 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:40.131214 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88s8w\" (UniqueName: \"kubernetes.io/projected/b0938f71-f1d8-4342-8d0a-7aed59adef39-kube-api-access-88s8w\") pod \"e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:20:40.135258 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:40.135042 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0938f71-f1d8-4342-8d0a-7aed59adef39-istio-token\") pod \"e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:20:40.262178 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:40.262117 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:20:40.611973 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:40.611849 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv"] Oct 30 07:20:40.615247 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:20:40.615218 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0938f71_f1d8_4342_8d0a_7aed59adef39.slice/crio-eaff35d589365094604fa017e4c11dea24d192717ace1e73715b812b7a61718a WatchSource:0}: Error finding container eaff35d589365094604fa017e4c11dea24d192717ace1e73715b812b7a61718a: Status 404 returned error can't find the container with id eaff35d589365094604fa017e4c11dea24d192717ace1e73715b812b7a61718a Oct 30 07:20:40.618463 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:40.618416 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:20:40.618567 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:40.618519 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:20:40.618631 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:40.618582 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:20:41.479254 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:41.479213 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" event={"ID":"b0938f71-f1d8-4342-8d0a-7aed59adef39","Type":"ContainerStarted","Data":"74378f4a114b21f52a0d256fd583040199effa222ac25fd48a58b1e50a660be2"} Oct 30 07:20:41.479730 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:41.479283 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" event={"ID":"b0938f71-f1d8-4342-8d0a-7aed59adef39","Type":"ContainerStarted","Data":"eaff35d589365094604fa017e4c11dea24d192717ace1e73715b812b7a61718a"} Oct 30 07:20:41.553064 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:41.552944 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" podStartSLOduration=2.552923946 podStartE2EDuration="2.552923946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:20:41.538123128 +0000 UTC m=+1447.881171110" watchObservedRunningTime="2026-10-30 07:20:41.552923946 +0000 UTC m=+1447.895971911" Oct 30 07:20:42.262628 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:42.262588 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:20:42.268994 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:42.268968 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:20:42.394407 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:42.394375 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn" Oct 30 07:20:42.487928 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:42.487899 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:20:42.489403 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:42.489371 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:20:42.727532 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:42.725071 4951 patch_prober.go:29] interesting pod/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": read tcp 10.132.0.2:58398->10.132.0.122:15021: read: connection reset by peer" start-of-body= Oct 30 07:20:42.727532 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:42.725138 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" podUID="0c51720b-1e0b-4cb0-a1e2-08065c91acde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": read tcp 10.132.0.2:58398->10.132.0.122:15021: read: connection reset by peer" Oct 30 07:20:43.623138 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.623097 4951 generic.go:356] "Generic (PLEG): container finished" podID="f3d3f735-6cd0-4153-a6ff-88f77f4941ca" containerID="e7c7a4790b3a792a5a717ead1f3d71eaa2577132870247c020633cd6e3c619a2" exitCode=0 Oct 30 07:20:43.623489 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.623253 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" event={"ID":"f3d3f735-6cd0-4153-a6ff-88f77f4941ca","Type":"ContainerDied","Data":"e7c7a4790b3a792a5a717ead1f3d71eaa2577132870247c020633cd6e3c619a2"} Oct 30 07:20:43.623489 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.623348 4951 scope.go:122] "RemoveContainer" containerID="27a16ce17a1d15ea1f14c838de0bb4c00094047037482fb1b05dab8f0bbebc87" Oct 30 07:20:43.626156 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.626133 4951 generic.go:356] "Generic (PLEG): container finished" podID="0c51720b-1e0b-4cb0-a1e2-08065c91acde" containerID="b34987ce0a1fefa04c782ce4dfe7aaaae0d8f38b4711817ea271e2d96088e65f" exitCode=0 Oct 30 07:20:43.626925 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.626890 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" event={"ID":"0c51720b-1e0b-4cb0-a1e2-08065c91acde","Type":"ContainerDied","Data":"b34987ce0a1fefa04c782ce4dfe7aaaae0d8f38b4711817ea271e2d96088e65f"} Oct 30 07:20:43.627336 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.627315 4951 scope.go:122] "RemoveContainer" containerID="b34987ce0a1fefa04c782ce4dfe7aaaae0d8f38b4711817ea271e2d96088e65f" Oct 30 07:20:43.629502 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.629475 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:20:43.629684 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.629666 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:20:43.629828 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.629814 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:20:43.689160 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.689131 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:20:43.808159 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.808050 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istio-token\" (UniqueName: \"kubernetes.io/projected/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istio-token\") pod \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " Oct 30 07:20:43.808359 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.808167 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istio-data\") pod \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " Oct 30 07:20:43.808359 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.808241 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istiod-ca-cert\") pod \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " Oct 30 07:20:43.808480 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.808357 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-credential-socket\") pod \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " Oct 30 07:20:43.808480 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.808454 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istio-podinfo\") pod \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " Oct 30 07:20:43.808601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.808487 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-kube-api-access-jqx8b\" (UniqueName: \"kubernetes.io/projected/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-kube-api-access-jqx8b\") pod \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " Oct 30 07:20:43.808601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.808545 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istio-envoy\") pod \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " Oct 30 07:20:43.809044 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.808599 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-workload-socket\") pod \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " Oct 30 07:20:43.809044 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.808652 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-workload-certs\") pod \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\" (UID: \"f3d3f735-6cd0-4153-a6ff-88f77f4941ca\") " Oct 30 07:20:43.809507 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.809482 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-workload-certs" pod "f3d3f735-6cd0-4153-a6ff-88f77f4941ca" (UID: "f3d3f735-6cd0-4153-a6ff-88f77f4941ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:20:43.814761 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.812930 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istiod-ca-cert" pod "f3d3f735-6cd0-4153-a6ff-88f77f4941ca" (UID: "f3d3f735-6cd0-4153-a6ff-88f77f4941ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:20:43.814761 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.813189 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istio-data" pod "f3d3f735-6cd0-4153-a6ff-88f77f4941ca" (UID: "f3d3f735-6cd0-4153-a6ff-88f77f4941ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:20:43.814761 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.813673 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-credential-socket" pod "f3d3f735-6cd0-4153-a6ff-88f77f4941ca" (UID: "f3d3f735-6cd0-4153-a6ff-88f77f4941ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:20:43.814761 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.813847 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-workload-socket" pod "f3d3f735-6cd0-4153-a6ff-88f77f4941ca" (UID: "f3d3f735-6cd0-4153-a6ff-88f77f4941ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:20:43.815799 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.815771 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istio-token" pod "f3d3f735-6cd0-4153-a6ff-88f77f4941ca" (UID: "f3d3f735-6cd0-4153-a6ff-88f77f4941ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:20:43.826255 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.819960 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istio-envoy" pod "f3d3f735-6cd0-4153-a6ff-88f77f4941ca" (UID: "f3d3f735-6cd0-4153-a6ff-88f77f4941ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:20:43.826255 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.823839 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istio-podinfo" pod "f3d3f735-6cd0-4153-a6ff-88f77f4941ca" (UID: "f3d3f735-6cd0-4153-a6ff-88f77f4941ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:20:43.826255 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.824205 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-kube-api-access-jqx8b" pod "f3d3f735-6cd0-4153-a6ff-88f77f4941ca" (UID: "f3d3f735-6cd0-4153-a6ff-88f77f4941ca"). InnerVolumeSpecName "kube-api-access-jqx8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:20:43.911654 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.911551 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:43.911883 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.911859 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:43.911983 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.911890 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:43.911983 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.911905 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:43.911983 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.911919 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:43.911983 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.911932 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:43.911983 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.911945 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:43.911983 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.911958 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:43.911983 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:43.911972 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqx8b\" (UniqueName: \"kubernetes.io/projected/f3d3f735-6cd0-4153-a6ff-88f77f4941ca-kube-api-access-jqx8b\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:44.662537 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:44.662503 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" event={"ID":"0c51720b-1e0b-4cb0-a1e2-08065c91acde","Type":"ContainerStarted","Data":"1cd7098c17f3398406adb9999819c2ead1525e521622cc70fa526329f51984e4"} Oct 30 07:20:44.664501 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:44.664470 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" Oct 30 07:20:44.664682 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:44.664515 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v" event={"ID":"f3d3f735-6cd0-4153-a6ff-88f77f4941ca","Type":"ContainerDied","Data":"8e70637230d66cb975a6362fab7643d6529641d214ef248fdd1490ffc1376224"} Oct 30 07:20:44.664682 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:44.664558 4951 scope.go:122] "RemoveContainer" containerID="e7c7a4790b3a792a5a717ead1f3d71eaa2577132870247c020633cd6e3c619a2" Oct 30 07:20:44.681598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:44.676895 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:20:44.681598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:44.677016 4951 patch_prober.go:29] interesting pod/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 07:20:44.681598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:44.677057 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" podUID="0c51720b-1e0b-4cb0-a1e2-08065c91acde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 07:20:44.787519 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:44.787479 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v"] Oct 30 07:20:44.793469 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:44.793434 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-486cd4-openshift-default-7b44878777-zq44v"] Oct 30 07:20:45.676528 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:45.676490 4951 patch_prober.go:29] interesting pod/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 07:20:45.677028 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:45.676539 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" podUID="0c51720b-1e0b-4cb0-a1e2-08065c91acde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 07:20:45.783513 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:45.783476 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3d3f735-6cd0-4153-a6ff-88f77f4941ca" path="/var/lib/kubelet/pods/f3d3f735-6cd0-4153-a6ff-88f77f4941ca/volumes" Oct 30 07:20:46.677244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:46.677208 4951 patch_prober.go:29] interesting pod/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 07:20:46.677764 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:46.677269 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" podUID="0c51720b-1e0b-4cb0-a1e2-08065c91acde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 07:20:47.321403 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:47.321369 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-59c44dfbbc-4c96d"] Oct 30 07:20:47.326593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:47.322378 4951 cpu_manager.go:414] "containerMap: removing container" podUID="f3d3f735-6cd0-4153-a6ff-88f77f4941ca" containerName="istio-proxy" Oct 30 07:20:47.326593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:47.322404 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3d3f735-6cd0-4153-a6ff-88f77f4941ca" containerName="istio-proxy" Oct 30 07:20:47.326593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:47.322418 4951 cpu_manager.go:414] "containerMap: removing container" podUID="f3d3f735-6cd0-4153-a6ff-88f77f4941ca" containerName="istio-proxy" Oct 30 07:20:47.326593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:47.322426 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3d3f735-6cd0-4153-a6ff-88f77f4941ca" containerName="istio-proxy" Oct 30 07:20:47.336605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:47.334800 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3d3f735-6cd0-4153-a6ff-88f77f4941ca" containerName="istio-proxy" Oct 30 07:20:47.336605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:47.334829 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3d3f735-6cd0-4153-a6ff-88f77f4941ca" containerName="istio-proxy" Oct 30 07:20:47.347596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:47.346836 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-59c44dfbbc-4c96d" Oct 30 07:20:47.353672 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:47.353640 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-59c44dfbbc-4c96d"] Oct 30 07:20:47.378714 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:47.378677 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-17bd11\"" Oct 30 07:20:47.378866 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:47.378814 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-17bd11-dockercfg-n77sp\"" Oct 30 07:20:47.429593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:47.418621 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pghm\" (UniqueName: \"kubernetes.io/projected/b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e-kube-api-access-5pghm\") pod \"payload-pre-processing-e2e-ipp-a-17bd11-59c44dfbbc-4c96d\" (UID: \"b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-59c44dfbbc-4c96d" Oct 30 07:20:47.429593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:47.418903 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-17bd11-59c44dfbbc-4c96d\" (UID: \"b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-59c44dfbbc-4c96d" Oct 30 07:20:47.569595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:47.566072 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-17bd11-59c44dfbbc-4c96d\" (UID: \"b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-59c44dfbbc-4c96d" Oct 30 07:20:47.569595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:47.566249 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pghm\" (UniqueName: \"kubernetes.io/projected/b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e-kube-api-access-5pghm\") pod \"payload-pre-processing-e2e-ipp-a-17bd11-59c44dfbbc-4c96d\" (UID: \"b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-59c44dfbbc-4c96d" Oct 30 07:20:47.621723 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:47.611833 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-17bd11-59c44dfbbc-4c96d\" (UID: \"b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-59c44dfbbc-4c96d" Oct 30 07:20:47.650638 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:47.643654 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pghm\" (UniqueName: \"kubernetes.io/projected/b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e-kube-api-access-5pghm\") pod \"payload-pre-processing-e2e-ipp-a-17bd11-59c44dfbbc-4c96d\" (UID: \"b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-59c44dfbbc-4c96d" Oct 30 07:20:47.885152 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:47.885053 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-59c44dfbbc-4c96d" Oct 30 07:20:47.885901 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:47.885794 4951 patch_prober.go:29] interesting pod/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 07:20:47.887342 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:47.887310 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" podUID="0c51720b-1e0b-4cb0-a1e2-08065c91acde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 07:20:47.906290 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:47.906263 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-17bd11-57864cfc76-zw5qj"] Oct 30 07:20:47.912813 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:47.912784 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-17bd11-57864cfc76-zw5qj"] Oct 30 07:20:47.912947 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:47.912927 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-57864cfc76-zw5qj" Oct 30 07:20:47.988733 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:47.988695 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e588a7d-d969-4313-aa99-54cd1776829c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-17bd11-57864cfc76-zw5qj\" (UID: \"7e588a7d-d969-4313-aa99-54cd1776829c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-57864cfc76-zw5qj" Oct 30 07:20:47.989041 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:47.989024 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xjtr\" (UniqueName: \"kubernetes.io/projected/7e588a7d-d969-4313-aa99-54cd1776829c-kube-api-access-4xjtr\") pod \"payload-processing-e2e-ipp-a-17bd11-57864cfc76-zw5qj\" (UID: \"7e588a7d-d969-4313-aa99-54cd1776829c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-57864cfc76-zw5qj" Oct 30 07:20:48.090702 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:48.090661 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xjtr\" (UniqueName: \"kubernetes.io/projected/7e588a7d-d969-4313-aa99-54cd1776829c-kube-api-access-4xjtr\") pod \"payload-processing-e2e-ipp-a-17bd11-57864cfc76-zw5qj\" (UID: \"7e588a7d-d969-4313-aa99-54cd1776829c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-57864cfc76-zw5qj" Oct 30 07:20:48.090868 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:48.090823 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e588a7d-d969-4313-aa99-54cd1776829c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-17bd11-57864cfc76-zw5qj\" (UID: \"7e588a7d-d969-4313-aa99-54cd1776829c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-57864cfc76-zw5qj" Oct 30 07:20:48.091790 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:48.091766 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e588a7d-d969-4313-aa99-54cd1776829c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-17bd11-57864cfc76-zw5qj\" (UID: \"7e588a7d-d969-4313-aa99-54cd1776829c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-57864cfc76-zw5qj" Oct 30 07:20:48.116043 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:48.116015 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xjtr\" (UniqueName: \"kubernetes.io/projected/7e588a7d-d969-4313-aa99-54cd1776829c-kube-api-access-4xjtr\") pod \"payload-processing-e2e-ipp-a-17bd11-57864cfc76-zw5qj\" (UID: \"7e588a7d-d969-4313-aa99-54cd1776829c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-57864cfc76-zw5qj" Oct 30 07:20:48.252915 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:48.252886 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-57864cfc76-zw5qj" Oct 30 07:20:48.253076 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:48.253050 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-59c44dfbbc-4c96d"] Oct 30 07:20:48.255705 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:20:48.255548 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a415ba_5c75_4ae9_bbf2_60c68eb8a53e.slice/crio-6b00f6ccf9c7f8d9e6cccc52119b545d6a7c9ca7244f1d56a831eb6d5391ae6d WatchSource:0}: Error finding container 6b00f6ccf9c7f8d9e6cccc52119b545d6a7c9ca7244f1d56a831eb6d5391ae6d: Status 404 returned error can't find the container with id 6b00f6ccf9c7f8d9e6cccc52119b545d6a7c9ca7244f1d56a831eb6d5391ae6d Oct 30 07:20:48.578739 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:48.578709 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-17bd11-57864cfc76-zw5qj"] Oct 30 07:20:48.579064 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:20:48.579042 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e588a7d_d969_4313_aa99_54cd1776829c.slice/crio-e8020700c918a939319bafe18bb668b821ffc1112c2d24368efa11294f7e0966 WatchSource:0}: Error finding container e8020700c918a939319bafe18bb668b821ffc1112c2d24368efa11294f7e0966: Status 404 returned error can't find the container with id e8020700c918a939319bafe18bb668b821ffc1112c2d24368efa11294f7e0966 Oct 30 07:20:48.676636 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:48.676605 4951 patch_prober.go:29] interesting pod/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 07:20:48.676760 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:48.676666 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" podUID="0c51720b-1e0b-4cb0-a1e2-08065c91acde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 07:20:48.829660 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:48.825978 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-59c44dfbbc-4c96d" event={"ID":"b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e","Type":"ContainerStarted","Data":"1c1dfc39d023b9a7b3f8d887ba72222881d768f6945894522c97b4a9cc87bfed"} Oct 30 07:20:48.829660 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:48.826050 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-59c44dfbbc-4c96d" event={"ID":"b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e","Type":"ContainerStarted","Data":"6b00f6ccf9c7f8d9e6cccc52119b545d6a7c9ca7244f1d56a831eb6d5391ae6d"} Oct 30 07:20:48.829660 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:48.827240 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-59c44dfbbc-4c96d" Oct 30 07:20:48.839592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:48.838334 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-57864cfc76-zw5qj" event={"ID":"7e588a7d-d969-4313-aa99-54cd1776829c","Type":"ContainerStarted","Data":"78d873a6b22881648f2054ab1865dcd83c557f7d4938d2fdcc6f295b74607387"} Oct 30 07:20:48.839592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:48.838393 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-57864cfc76-zw5qj" event={"ID":"7e588a7d-d969-4313-aa99-54cd1776829c","Type":"ContainerStarted","Data":"e8020700c918a939319bafe18bb668b821ffc1112c2d24368efa11294f7e0966"} Oct 30 07:20:48.839592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:48.839210 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-57864cfc76-zw5qj" Oct 30 07:20:48.870402 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:48.870344 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-59c44dfbbc-4c96d" podStartSLOduration=1.870327428 podStartE2EDuration="1.870327428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:20:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:20:48.868033157 +0000 UTC m=+1455.211081141" watchObservedRunningTime="2026-10-30 07:20:48.870327428 +0000 UTC m=+1455.213375394" Oct 30 07:20:48.910183 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:48.910127 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-57864cfc76-zw5qj" podStartSLOduration=1.910109337 podStartE2EDuration="1.910109337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:20:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:20:48.907831494 +0000 UTC m=+1455.250879460" watchObservedRunningTime="2026-10-30 07:20:48.910109337 +0000 UTC m=+1455.253157303" Oct 30 07:20:49.677627 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:49.677548 4951 patch_prober.go:29] interesting pod/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 07:20:49.677827 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:49.677649 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" podUID="0c51720b-1e0b-4cb0-a1e2-08065c91acde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 07:20:50.681011 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:50.680971 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:20:50.681488 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:50.681451 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:20:50.682246 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:50.682223 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:20:50.844037 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:50.843995 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-052254-bc7487fc6-qw9h7"] Oct 30 07:20:50.844249 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:50.844224 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-052254-bc7487fc6-qw9h7" podUID="0fdb2170-1155-429b-87e9-6827bdafbdee" containerName="maas-api" containerID="cri-o://b0fa91ca01bdb115c03247a3d4783559fca221fbaf870be90c32d0147e26e820" gracePeriod=30 Oct 30 07:20:51.489740 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:51.489695 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-052254-bc7487fc6-qw9h7" Oct 30 07:20:51.553210 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:51.553168 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fdb2170-1155-429b-87e9-6827bdafbdee-kube-api-access-jm654\" (UniqueName: \"kubernetes.io/projected/0fdb2170-1155-429b-87e9-6827bdafbdee-kube-api-access-jm654\") pod \"0fdb2170-1155-429b-87e9-6827bdafbdee\" (UID: \"0fdb2170-1155-429b-87e9-6827bdafbdee\") " Oct 30 07:20:51.553402 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:51.553258 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fdb2170-1155-429b-87e9-6827bdafbdee-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fdb2170-1155-429b-87e9-6827bdafbdee-openshift-service-ca\") pod \"0fdb2170-1155-429b-87e9-6827bdafbdee\" (UID: \"0fdb2170-1155-429b-87e9-6827bdafbdee\") " Oct 30 07:20:51.553402 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:51.553359 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fdb2170-1155-429b-87e9-6827bdafbdee-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fdb2170-1155-429b-87e9-6827bdafbdee-metrics-certs\") pod \"0fdb2170-1155-429b-87e9-6827bdafbdee\" (UID: \"0fdb2170-1155-429b-87e9-6827bdafbdee\") " Oct 30 07:20:51.553530 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:51.553459 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fdb2170-1155-429b-87e9-6827bdafbdee-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fdb2170-1155-429b-87e9-6827bdafbdee-maas-api-tls\") pod \"0fdb2170-1155-429b-87e9-6827bdafbdee\" (UID: \"0fdb2170-1155-429b-87e9-6827bdafbdee\") " Oct 30 07:20:51.555233 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:51.555204 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fdb2170-1155-429b-87e9-6827bdafbdee-openshift-service-ca" pod "0fdb2170-1155-429b-87e9-6827bdafbdee" (UID: "0fdb2170-1155-429b-87e9-6827bdafbdee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:20:51.560648 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:51.559169 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fdb2170-1155-429b-87e9-6827bdafbdee-metrics-certs" pod "0fdb2170-1155-429b-87e9-6827bdafbdee" (UID: "0fdb2170-1155-429b-87e9-6827bdafbdee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:20:51.560648 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:51.559336 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fdb2170-1155-429b-87e9-6827bdafbdee-maas-api-tls" pod "0fdb2170-1155-429b-87e9-6827bdafbdee" (UID: "0fdb2170-1155-429b-87e9-6827bdafbdee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:20:51.560852 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:51.560810 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fdb2170-1155-429b-87e9-6827bdafbdee-kube-api-access-jm654" pod "0fdb2170-1155-429b-87e9-6827bdafbdee" (UID: "0fdb2170-1155-429b-87e9-6827bdafbdee"). InnerVolumeSpecName "kube-api-access-jm654". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:20:51.655397 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:51.655290 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fdb2170-1155-429b-87e9-6827bdafbdee-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:51.655397 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:51.655326 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm654\" (UniqueName: \"kubernetes.io/projected/0fdb2170-1155-429b-87e9-6827bdafbdee-kube-api-access-jm654\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:51.655397 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:51.655338 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fdb2170-1155-429b-87e9-6827bdafbdee-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:51.655397 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:51.655346 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fdb2170-1155-429b-87e9-6827bdafbdee-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:20:51.863082 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:51.863044 4951 generic.go:356] "Generic (PLEG): container finished" podID="0fdb2170-1155-429b-87e9-6827bdafbdee" containerID="b0fa91ca01bdb115c03247a3d4783559fca221fbaf870be90c32d0147e26e820" exitCode=0 Oct 30 07:20:51.863539 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:51.863092 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-052254-bc7487fc6-qw9h7" Oct 30 07:20:51.863539 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:51.863092 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-052254-bc7487fc6-qw9h7" event={"ID":"0fdb2170-1155-429b-87e9-6827bdafbdee","Type":"ContainerDied","Data":"b0fa91ca01bdb115c03247a3d4783559fca221fbaf870be90c32d0147e26e820"} Oct 30 07:20:51.863539 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:51.863169 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-052254-bc7487fc6-qw9h7" event={"ID":"0fdb2170-1155-429b-87e9-6827bdafbdee","Type":"ContainerDied","Data":"1024fd33d20ca81b62911d20e9e0ebb09c142f0cbc3677ec39a6d6ee3f40b0b3"} Oct 30 07:20:51.863539 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:51.863203 4951 scope.go:122] "RemoveContainer" containerID="b0fa91ca01bdb115c03247a3d4783559fca221fbaf870be90c32d0147e26e820" Oct 30 07:20:51.883824 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:51.883805 4951 scope.go:122] "RemoveContainer" containerID="b0fa91ca01bdb115c03247a3d4783559fca221fbaf870be90c32d0147e26e820" Oct 30 07:20:51.884118 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:20:51.884093 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0fa91ca01bdb115c03247a3d4783559fca221fbaf870be90c32d0147e26e820\": container with ID starting with b0fa91ca01bdb115c03247a3d4783559fca221fbaf870be90c32d0147e26e820 not found: ID does not exist" containerID="b0fa91ca01bdb115c03247a3d4783559fca221fbaf870be90c32d0147e26e820" Oct 30 07:20:51.884197 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:51.884129 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0fa91ca01bdb115c03247a3d4783559fca221fbaf870be90c32d0147e26e820"} err="failed to get container status \"b0fa91ca01bdb115c03247a3d4783559fca221fbaf870be90c32d0147e26e820\": rpc error: code = NotFound desc = could not find container \"b0fa91ca01bdb115c03247a3d4783559fca221fbaf870be90c32d0147e26e820\": container with ID starting with b0fa91ca01bdb115c03247a3d4783559fca221fbaf870be90c32d0147e26e820 not found: ID does not exist" Oct 30 07:20:51.916732 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:51.916673 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-052254-bc7487fc6-qw9h7"] Oct 30 07:20:51.930482 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:51.930451 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-052254-bc7487fc6-qw9h7"] Oct 30 07:20:53.025170 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:20:53.025130 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:20:53.774751 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:53.774715 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fdb2170-1155-429b-87e9-6827bdafbdee" path="/var/lib/kubelet/pods/0fdb2170-1155-429b-87e9-6827bdafbdee/volumes" Oct 30 07:20:59.617606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:59.572244 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2"] Oct 30 07:20:59.630640 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:20:59.620659 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" podUID="94280cc1-4ffe-4862-9cfc-1d61949679ed" containerName="istio-proxy" containerID="cri-o://99b7e757f1676994d8b02862bbf67ac99c2a1a2d418d81dfaf8f51e991653ec1" gracePeriod=30 Oct 30 07:21:00.043752 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:00.043708 4951 patch_prober.go:29] interesting pod/e2e-api-b-052254-openshift-default-57b897c995-9hsd2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:21:00.044057 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:00.044028 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" podUID="94280cc1-4ffe-4862-9cfc-1d61949679ed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:21:00.883833 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:00.883801 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-59c44dfbbc-4c96d" Oct 30 07:21:00.915786 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:00.904549 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-57864cfc76-zw5qj" Oct 30 07:21:01.109598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.109006 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn"] Oct 30 07:21:01.115767 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.110170 4951 cpu_manager.go:414] "containerMap: removing container" podUID="0fdb2170-1155-429b-87e9-6827bdafbdee" containerName="maas-api" Oct 30 07:21:01.115767 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.110199 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fdb2170-1155-429b-87e9-6827bdafbdee" containerName="maas-api" Oct 30 07:21:01.115767 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.110372 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fdb2170-1155-429b-87e9-6827bdafbdee" containerName="maas-api" Oct 30 07:21:01.118988 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.118962 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:21:01.145606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.145496 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-istio-data\") pod \"e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:21:01.145840 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.145820 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-istio-envoy\") pod \"e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:21:01.145979 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.145964 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-credential-socket\") pod \"e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:21:01.146104 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.146092 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ed0bc5e-986d-4699-9513-053f7fe24b65-istio-token\") pod \"e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:21:01.146210 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.146197 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ed0bc5e-986d-4699-9513-053f7fe24b65-istiod-ca-cert\") pod \"e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:21:01.146367 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.146352 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-workload-socket\") pod \"e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:21:01.146490 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.146478 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-workload-certs\") pod \"e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:21:01.146601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.146564 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ed0bc5e-986d-4699-9513-053f7fe24b65-istio-podinfo\") pod \"e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:21:01.146786 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.146769 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h79x\" (UniqueName: \"kubernetes.io/projected/5ed0bc5e-986d-4699-9513-053f7fe24b65-kube-api-access-6h79x\") pod \"e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:21:01.147150 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.146366 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-fc546d-openshift-default-dockercfg-gk4dg\"" Oct 30 07:21:01.215516 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.215477 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn"] Oct 30 07:21:01.297223 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.297184 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-workload-socket\") pod \"e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:21:01.298690 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.298664 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-workload-certs\") pod \"e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:21:01.299183 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.299165 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ed0bc5e-986d-4699-9513-053f7fe24b65-istio-podinfo\") pod \"e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:21:01.300098 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.300077 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h79x\" (UniqueName: \"kubernetes.io/projected/5ed0bc5e-986d-4699-9513-053f7fe24b65-kube-api-access-6h79x\") pod \"e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:21:01.300684 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.300662 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-istio-data\") pod \"e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:21:01.301109 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.301092 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-istio-envoy\") pod \"e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:21:01.301620 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.301601 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-credential-socket\") pod \"e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:21:01.302226 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.302207 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ed0bc5e-986d-4699-9513-053f7fe24b65-istio-token\") pod \"e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:21:01.422644 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.402939 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ed0bc5e-986d-4699-9513-053f7fe24b65-istiod-ca-cert\") pod \"e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:21:01.422644 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.302118 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-credential-socket\") pod \"e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:21:01.422644 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.298552 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-workload-socket\") pod \"e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:21:01.422644 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.400857 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-579b5d699b-7g76s"] Oct 30 07:21:01.422644 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.299110 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-workload-certs\") pod \"e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:21:01.449594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.301041 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-istio-data\") pod \"e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:21:01.482759 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.482723 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-579b5d699b-7g76s" Oct 30 07:21:01.557601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.553774 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-579b5d699b-7g76s"] Oct 30 07:21:01.604597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.579850 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ab3d83\"" Oct 30 07:21:01.604597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.587037 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h79x\" (UniqueName: \"kubernetes.io/projected/5ed0bc5e-986d-4699-9513-053f7fe24b65-kube-api-access-6h79x\") pod \"e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:21:01.728667 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.725804 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b26b6b9-3f18-4090-b0e9-d2d537a03954-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ab3d83-579b5d699b-7g76s\" (UID: \"0b26b6b9-3f18-4090-b0e9-d2d537a03954\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-579b5d699b-7g76s" Oct 30 07:21:01.728667 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.725891 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4frsv\" (UniqueName: \"kubernetes.io/projected/0b26b6b9-3f18-4090-b0e9-d2d537a03954-kube-api-access-4frsv\") pod \"payload-processing-e2e-ipp-a-ab3d83-579b5d699b-7g76s\" (UID: \"0b26b6b9-3f18-4090-b0e9-d2d537a03954\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-579b5d699b-7g76s" Oct 30 07:21:01.753600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.747408 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ab3d83-dockercfg-c7rtp\"" Oct 30 07:21:01.803697 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.760360 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ed0bc5e-986d-4699-9513-053f7fe24b65-istio-podinfo\") pod \"e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:21:01.803697 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.793916 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ed0bc5e-986d-4699-9513-053f7fe24b65-istiod-ca-cert\") pod \"e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:21:01.803956 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.794190 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-istio-envoy\") pod \"e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:21:01.803956 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.794536 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ed0bc5e-986d-4699-9513-053f7fe24b65-istio-token\") pod \"e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:21:01.804054 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.804018 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:21:01.881594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.876686 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b26b6b9-3f18-4090-b0e9-d2d537a03954-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ab3d83-579b5d699b-7g76s\" (UID: \"0b26b6b9-3f18-4090-b0e9-d2d537a03954\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-579b5d699b-7g76s" Oct 30 07:21:01.883039 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.883012 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4frsv\" (UniqueName: \"kubernetes.io/projected/0b26b6b9-3f18-4090-b0e9-d2d537a03954-kube-api-access-4frsv\") pod \"payload-processing-e2e-ipp-a-ab3d83-579b5d699b-7g76s\" (UID: \"0b26b6b9-3f18-4090-b0e9-d2d537a03954\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-579b5d699b-7g76s" Oct 30 07:21:01.884412 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.883251 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b26b6b9-3f18-4090-b0e9-d2d537a03954-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ab3d83-579b5d699b-7g76s\" (UID: \"0b26b6b9-3f18-4090-b0e9-d2d537a03954\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-579b5d699b-7g76s" Oct 30 07:21:01.906371 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.906342 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4frsv\" (UniqueName: \"kubernetes.io/projected/0b26b6b9-3f18-4090-b0e9-d2d537a03954-kube-api-access-4frsv\") pod \"payload-processing-e2e-ipp-a-ab3d83-579b5d699b-7g76s\" (UID: \"0b26b6b9-3f18-4090-b0e9-d2d537a03954\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-579b5d699b-7g76s" Oct 30 07:21:01.978479 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.977626 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3d2270-5db8699bbb-56xv9"] Oct 30 07:21:01.999639 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:01.987419 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-655945b85f-qhdlr"] Oct 30 07:21:02.119182 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:02.115162 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-579b5d699b-7g76s" Oct 30 07:21:02.298633 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:02.298593 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn"] Oct 30 07:21:02.304145 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:02.304087 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:21:02.304265 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:02.304189 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:21:02.304265 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:02.304249 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:21:02.859593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:02.854381 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-579b5d699b-7g76s"] Oct 30 07:21:03.313513 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:03.313478 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-579b5d699b-7g76s" event={"ID":"0b26b6b9-3f18-4090-b0e9-d2d537a03954","Type":"ContainerStarted","Data":"1b80f6924bf55e531f38a4de2ce7922e8590e7abf0e60ebca61500c58851f523"} Oct 30 07:21:03.354607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:03.354237 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" event={"ID":"5ed0bc5e-986d-4699-9513-053f7fe24b65","Type":"ContainerStarted","Data":"65a1c50f6baeb046dd16c32d68a885d7cd6561ceb3282bb042828795c92cd695"} Oct 30 07:21:03.498003 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:03.491441 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" podStartSLOduration=2.491418685 podStartE2EDuration="2.491418685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:21:03.434293023 +0000 UTC m=+1469.777340988" watchObservedRunningTime="2026-10-30 07:21:03.491418685 +0000 UTC m=+1469.834466650" Oct 30 07:21:03.856595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:03.851587 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:21:04.389272 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:04.389236 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-579b5d699b-7g76s" event={"ID":"0b26b6b9-3f18-4090-b0e9-d2d537a03954","Type":"ContainerStarted","Data":"4ea2f88384f4b461bbe2a0b1532a712a60fe8cf7c92b9c87f2d1de170b754e82"} Oct 30 07:21:04.399959 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:04.399932 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-579b5d699b-7g76s" Oct 30 07:21:04.458191 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:04.457182 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" event={"ID":"5ed0bc5e-986d-4699-9513-053f7fe24b65","Type":"ContainerStarted","Data":"800de26dd48119553231546144674293d0dbf81bdbd6ecf9c935135284d2aabb"} Oct 30 07:21:04.473411 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:04.473352 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-579b5d699b-7g76s" podStartSLOduration=3.4733317550000002 podStartE2EDuration="3.473331755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:21:04.456312775 +0000 UTC m=+1470.799360790" watchObservedRunningTime="2026-10-30 07:21:04.473331755 +0000 UTC m=+1470.816379721" Oct 30 07:21:04.812662 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:04.812271 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:21:05.484889 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:05.483531 4951 generic.go:356] "Generic (PLEG): container finished" podID="94280cc1-4ffe-4862-9cfc-1d61949679ed" containerID="99b7e757f1676994d8b02862bbf67ac99c2a1a2d418d81dfaf8f51e991653ec1" exitCode=0 Oct 30 07:21:05.484889 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:05.484490 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" event={"ID":"94280cc1-4ffe-4862-9cfc-1d61949679ed","Type":"ContainerDied","Data":"99b7e757f1676994d8b02862bbf67ac99c2a1a2d418d81dfaf8f51e991653ec1"} Oct 30 07:21:05.484889 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:05.484606 4951 scope.go:122] "RemoveContainer" containerID="f10083df04ab335b726acd7c22e316dd8cabbf4aabd50511ecd7911f40f4c291" Oct 30 07:21:05.494393 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:05.494361 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:21:05.514116 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:05.510746 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:21:05.700740 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:05.700699 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:21:05.812115 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:05.812082 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94280cc1-4ffe-4862-9cfc-1d61949679ed-istio-token\" (UniqueName: \"kubernetes.io/projected/94280cc1-4ffe-4862-9cfc-1d61949679ed-istio-token\") pod \"94280cc1-4ffe-4862-9cfc-1d61949679ed\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " Oct 30 07:21:05.812708 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:05.812678 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-workload-certs\") pod \"94280cc1-4ffe-4862-9cfc-1d61949679ed\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " Oct 30 07:21:05.812990 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:05.812967 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-workload-certs" pod "94280cc1-4ffe-4862-9cfc-1d61949679ed" (UID: "94280cc1-4ffe-4862-9cfc-1d61949679ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:21:05.813089 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:05.813047 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-credential-socket\") pod \"94280cc1-4ffe-4862-9cfc-1d61949679ed\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " Oct 30 07:21:05.813089 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:05.813080 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94280cc1-4ffe-4862-9cfc-1d61949679ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94280cc1-4ffe-4862-9cfc-1d61949679ed-istiod-ca-cert\") pod \"94280cc1-4ffe-4862-9cfc-1d61949679ed\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " Oct 30 07:21:05.813210 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:05.813152 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94280cc1-4ffe-4862-9cfc-1d61949679ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94280cc1-4ffe-4862-9cfc-1d61949679ed-istio-podinfo\") pod \"94280cc1-4ffe-4862-9cfc-1d61949679ed\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " Oct 30 07:21:05.813210 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:05.813194 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-istio-data\") pod \"94280cc1-4ffe-4862-9cfc-1d61949679ed\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " Oct 30 07:21:05.813313 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:05.813251 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94280cc1-4ffe-4862-9cfc-1d61949679ed-kube-api-access-hdn8d\" (UniqueName: \"kubernetes.io/projected/94280cc1-4ffe-4862-9cfc-1d61949679ed-kube-api-access-hdn8d\") pod \"94280cc1-4ffe-4862-9cfc-1d61949679ed\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " Oct 30 07:21:05.813368 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:05.813324 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-workload-socket\") pod \"94280cc1-4ffe-4862-9cfc-1d61949679ed\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " Oct 30 07:21:05.813368 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:05.813362 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-istio-envoy\") pod \"94280cc1-4ffe-4862-9cfc-1d61949679ed\" (UID: \"94280cc1-4ffe-4862-9cfc-1d61949679ed\") " Oct 30 07:21:05.814944 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:05.814913 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94280cc1-4ffe-4862-9cfc-1d61949679ed-istiod-ca-cert" pod "94280cc1-4ffe-4862-9cfc-1d61949679ed" (UID: "94280cc1-4ffe-4862-9cfc-1d61949679ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:21:05.819029 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:05.815209 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-credential-socket" pod "94280cc1-4ffe-4862-9cfc-1d61949679ed" (UID: "94280cc1-4ffe-4862-9cfc-1d61949679ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:21:05.819029 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:05.815488 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-istio-data" pod "94280cc1-4ffe-4862-9cfc-1d61949679ed" (UID: "94280cc1-4ffe-4862-9cfc-1d61949679ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:21:05.819029 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:05.815716 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-workload-socket" pod "94280cc1-4ffe-4862-9cfc-1d61949679ed" (UID: "94280cc1-4ffe-4862-9cfc-1d61949679ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:21:05.819029 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:05.816157 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.819029 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:05.816179 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.819029 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:05.816194 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94280cc1-4ffe-4862-9cfc-1d61949679ed-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.819029 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:05.816207 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.819029 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:05.816220 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.858048 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:05.857943 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94280cc1-4ffe-4862-9cfc-1d61949679ed-kube-api-access-hdn8d" pod "94280cc1-4ffe-4862-9cfc-1d61949679ed" (UID: "94280cc1-4ffe-4862-9cfc-1d61949679ed"). InnerVolumeSpecName "kube-api-access-hdn8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:21:05.861272 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:05.861232 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94280cc1-4ffe-4862-9cfc-1d61949679ed-istio-token" pod "94280cc1-4ffe-4862-9cfc-1d61949679ed" (UID: "94280cc1-4ffe-4862-9cfc-1d61949679ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:21:05.861517 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:05.861488 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94280cc1-4ffe-4862-9cfc-1d61949679ed-istio-podinfo" pod "94280cc1-4ffe-4862-9cfc-1d61949679ed" (UID: "94280cc1-4ffe-4862-9cfc-1d61949679ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:21:05.876162 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:05.876113 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-istio-envoy" pod "94280cc1-4ffe-4862-9cfc-1d61949679ed" (UID: "94280cc1-4ffe-4862-9cfc-1d61949679ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:21:05.918995 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:05.918965 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94280cc1-4ffe-4862-9cfc-1d61949679ed-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.918995 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:05.918995 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94280cc1-4ffe-4862-9cfc-1d61949679ed-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.919228 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:05.919007 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94280cc1-4ffe-4862-9cfc-1d61949679ed-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.919228 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:05.919020 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdn8d\" (UniqueName: \"kubernetes.io/projected/94280cc1-4ffe-4862-9cfc-1d61949679ed-kube-api-access-hdn8d\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:21:06.496596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:06.495711 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" Oct 30 07:21:06.496596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:06.495760 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2" event={"ID":"94280cc1-4ffe-4862-9cfc-1d61949679ed","Type":"ContainerDied","Data":"37199ba5973ffbc8981500527ddd983e2cc49118cfa8bd242e6513ac43966396"} Oct 30 07:21:06.496596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:06.495829 4951 scope.go:122] "RemoveContainer" containerID="99b7e757f1676994d8b02862bbf67ac99c2a1a2d418d81dfaf8f51e991653ec1" Oct 30 07:21:06.560305 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:06.558642 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2"] Oct 30 07:21:06.571626 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:06.570274 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-052254-openshift-default-57b897c995-9hsd2"] Oct 30 07:21:06.978690 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:06.978653 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-5db8699bbb-56xv9" podUID="91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e" containerName="payload-processing" containerID="cri-o://3a1dd9ad5b4845566d5e0e829f030f7281696af57881ddc51cfe7fac0b26a923" gracePeriod=25 Oct 30 07:21:07.000426 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:07.000368 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-655945b85f-qhdlr" podUID="8889754c-55d5-4722-94dc-9579494fdd3e" containerName="payload-pre-processing" containerID="cri-o://70b63a6cb11f9ccb19c5b25baf2cdb3d18621666ba8ba5f88b84d42c3fa4e505" gracePeriod=25 Oct 30 07:21:07.496000 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:07.495968 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-5db8699bbb-56xv9" Oct 30 07:21:07.509166 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:07.508808 4951 generic.go:356] "Generic (PLEG): container finished" podID="91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e" containerID="3a1dd9ad5b4845566d5e0e829f030f7281696af57881ddc51cfe7fac0b26a923" exitCode=0 Oct 30 07:21:07.509166 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:07.508916 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-5db8699bbb-56xv9" Oct 30 07:21:07.509166 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:07.509058 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-5db8699bbb-56xv9" event={"ID":"91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e","Type":"ContainerDied","Data":"3a1dd9ad5b4845566d5e0e829f030f7281696af57881ddc51cfe7fac0b26a923"} Oct 30 07:21:07.509166 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:07.509092 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-5db8699bbb-56xv9" event={"ID":"91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e","Type":"ContainerDied","Data":"68172bfd1708401c6b22568630e31a5e0d805cdb260fa0535ad6254e0ce64bbf"} Oct 30 07:21:07.509166 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:07.509114 4951 scope.go:122] "RemoveContainer" containerID="3a1dd9ad5b4845566d5e0e829f030f7281696af57881ddc51cfe7fac0b26a923" Oct 30 07:21:07.512870 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:07.512458 4951 generic.go:356] "Generic (PLEG): container finished" podID="8889754c-55d5-4722-94dc-9579494fdd3e" containerID="70b63a6cb11f9ccb19c5b25baf2cdb3d18621666ba8ba5f88b84d42c3fa4e505" exitCode=0 Oct 30 07:21:07.512870 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:07.512561 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-655945b85f-qhdlr" event={"ID":"8889754c-55d5-4722-94dc-9579494fdd3e","Type":"ContainerDied","Data":"70b63a6cb11f9ccb19c5b25baf2cdb3d18621666ba8ba5f88b84d42c3fa4e505"} Oct 30 07:21:07.522438 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:07.522419 4951 scope.go:122] "RemoveContainer" containerID="3a1dd9ad5b4845566d5e0e829f030f7281696af57881ddc51cfe7fac0b26a923" Oct 30 07:21:07.522812 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:21:07.522770 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a1dd9ad5b4845566d5e0e829f030f7281696af57881ddc51cfe7fac0b26a923\": container with ID starting with 3a1dd9ad5b4845566d5e0e829f030f7281696af57881ddc51cfe7fac0b26a923 not found: ID does not exist" containerID="3a1dd9ad5b4845566d5e0e829f030f7281696af57881ddc51cfe7fac0b26a923" Oct 30 07:21:07.522963 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:07.522805 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a1dd9ad5b4845566d5e0e829f030f7281696af57881ddc51cfe7fac0b26a923"} err="failed to get container status \"3a1dd9ad5b4845566d5e0e829f030f7281696af57881ddc51cfe7fac0b26a923\": rpc error: code = NotFound desc = could not find container \"3a1dd9ad5b4845566d5e0e829f030f7281696af57881ddc51cfe7fac0b26a923\": container with ID starting with 3a1dd9ad5b4845566d5e0e829f030f7281696af57881ddc51cfe7fac0b26a923 not found: ID does not exist" Oct 30 07:21:07.587819 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:07.587753 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-655945b85f-qhdlr" Oct 30 07:21:07.595201 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:07.595158 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8889754c-55d5-4722-94dc-9579494fdd3e-kube-api-access-9jgkr\" (UniqueName: \"kubernetes.io/projected/8889754c-55d5-4722-94dc-9579494fdd3e-kube-api-access-9jgkr\") pod \"8889754c-55d5-4722-94dc-9579494fdd3e\" (UID: \"8889754c-55d5-4722-94dc-9579494fdd3e\") " Oct 30 07:21:07.595367 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:07.595304 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e-plugins-config-volume\") pod \"91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e\" (UID: \"91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e\") " Oct 30 07:21:07.595488 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:07.595470 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e-kube-api-access-vpxmm\" (UniqueName: \"kubernetes.io/projected/91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e-kube-api-access-vpxmm\") pod \"91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e\" (UID: \"91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e\") " Oct 30 07:21:07.595548 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:07.595532 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8889754c-55d5-4722-94dc-9579494fdd3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8889754c-55d5-4722-94dc-9579494fdd3e-plugins-config-volume\") pod \"8889754c-55d5-4722-94dc-9579494fdd3e\" (UID: \"8889754c-55d5-4722-94dc-9579494fdd3e\") " Oct 30 07:21:07.597823 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:07.597220 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8889754c-55d5-4722-94dc-9579494fdd3e-plugins-config-volume" pod "8889754c-55d5-4722-94dc-9579494fdd3e" (UID: "8889754c-55d5-4722-94dc-9579494fdd3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:21:07.597963 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:07.597875 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e-plugins-config-volume" pod "91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e" (UID: "91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:21:07.600394 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:07.600355 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8889754c-55d5-4722-94dc-9579494fdd3e-kube-api-access-9jgkr" pod "8889754c-55d5-4722-94dc-9579494fdd3e" (UID: "8889754c-55d5-4722-94dc-9579494fdd3e"). InnerVolumeSpecName "kube-api-access-9jgkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:21:07.600716 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:07.600691 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e-kube-api-access-vpxmm" pod "91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e" (UID: "91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e"). InnerVolumeSpecName "kube-api-access-vpxmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:21:07.704002 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:07.703959 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8889754c-55d5-4722-94dc-9579494fdd3e-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:21:07.704002 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:07.704002 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jgkr\" (UniqueName: \"kubernetes.io/projected/8889754c-55d5-4722-94dc-9579494fdd3e-kube-api-access-9jgkr\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:21:07.704277 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:07.704021 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:21:07.704277 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:07.704043 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpxmm\" (UniqueName: \"kubernetes.io/projected/91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e-kube-api-access-vpxmm\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:21:07.774194 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:07.774150 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94280cc1-4ffe-4862-9cfc-1d61949679ed" path="/var/lib/kubelet/pods/94280cc1-4ffe-4862-9cfc-1d61949679ed/volumes" Oct 30 07:21:07.851543 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:07.851443 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3d2270-5db8699bbb-56xv9"] Oct 30 07:21:07.862444 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:07.862405 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3d2270-5db8699bbb-56xv9"] Oct 30 07:21:08.071787 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:08.071734 4951 patch_prober.go:29] interesting pod/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:21:08.071998 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:08.071813 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" podUID="5c1b682b-01eb-4717-9184-d81851ade4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:21:08.525832 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:08.525664 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-655945b85f-qhdlr" event={"ID":"8889754c-55d5-4722-94dc-9579494fdd3e","Type":"ContainerDied","Data":"ad26a6efbfa5d60cb24d2ba8e22c23a4cdfab84854c3c9ab5de7dcd02f38af1e"} Oct 30 07:21:08.525832 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:08.525740 4951 scope.go:122] "RemoveContainer" containerID="70b63a6cb11f9ccb19c5b25baf2cdb3d18621666ba8ba5f88b84d42c3fa4e505" Oct 30 07:21:08.528824 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:08.528793 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-655945b85f-qhdlr" Oct 30 07:21:08.587909 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:08.587863 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-655945b85f-qhdlr"] Oct 30 07:21:08.591415 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:08.591378 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-655945b85f-qhdlr"] Oct 30 07:21:09.808147 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:09.808108 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8889754c-55d5-4722-94dc-9579494fdd3e" path="/var/lib/kubelet/pods/8889754c-55d5-4722-94dc-9579494fdd3e/volumes" Oct 30 07:21:09.809481 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:09.809445 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e" path="/var/lib/kubelet/pods/91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e/volumes" Oct 30 07:21:11.578045 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:11.578014 4951 generic.go:356] "Generic (PLEG): container finished" podID="9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b" containerID="893fe6868772fd31a178a7746a5974b29097011e48dbc7c66004b9e385ce56e5" exitCode=0 Oct 30 07:21:11.578499 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:11.578087 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" event={"ID":"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b","Type":"ContainerDied","Data":"893fe6868772fd31a178a7746a5974b29097011e48dbc7c66004b9e385ce56e5"} Oct 30 07:21:11.578634 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:11.578610 4951 scope.go:122] "RemoveContainer" containerID="893fe6868772fd31a178a7746a5974b29097011e48dbc7c66004b9e385ce56e5" Oct 30 07:21:11.579977 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:11.579957 4951 generic.go:356] "Generic (PLEG): container finished" podID="5c1b682b-01eb-4717-9184-d81851ade4b0" containerID="fa2c7640c0bc198242c773e7da1f563fb2c1ccd7fbeec578cf012036992e4450" exitCode=0 Oct 30 07:21:11.580109 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:11.580088 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" event={"ID":"5c1b682b-01eb-4717-9184-d81851ade4b0","Type":"ContainerDied","Data":"fa2c7640c0bc198242c773e7da1f563fb2c1ccd7fbeec578cf012036992e4450"} Oct 30 07:21:11.580319 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:11.580292 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:21:11.580383 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:11.580370 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:21:11.580431 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:11.580421 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:21:11.580535 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:11.580518 4951 scope.go:122] "RemoveContainer" containerID="fa2c7640c0bc198242c773e7da1f563fb2c1ccd7fbeec578cf012036992e4450" Oct 30 07:21:11.582114 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:11.582083 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:21:11.582200 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:11.582162 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:21:11.582239 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:11.582209 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:21:12.594786 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:12.594738 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" event={"ID":"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b","Type":"ContainerStarted","Data":"410542a5625835bd790467d2621db81aad747f62a19df3743a09aa8d92c69071"} Oct 30 07:21:12.597654 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:12.597623 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" event={"ID":"5c1b682b-01eb-4717-9184-d81851ade4b0","Type":"ContainerStarted","Data":"3e52bfcd6536d75017df91674af8f7417a7a20b386b9711fe8361a4bd8676c15"} Oct 30 07:21:13.008665 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:13.008638 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:21:13.008990 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:13.008965 4951 patch_prober.go:29] interesting pod/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 07:21:13.009045 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:13.009019 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" podUID="5c1b682b-01eb-4717-9184-d81851ade4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 07:21:13.192598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:13.191522 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:21:13.192598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:13.191677 4951 patch_prober.go:29] interesting pod/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 07:21:13.192598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:13.191745 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" podUID="9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 07:21:14.010408 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:14.010377 4951 patch_prober.go:29] interesting pod/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 07:21:14.010800 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:14.010425 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" podUID="5c1b682b-01eb-4717-9184-d81851ade4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 07:21:14.190023 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:14.189981 4951 patch_prober.go:29] interesting pod/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 07:21:14.190212 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:14.190038 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" podUID="9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 07:21:14.536518 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:14.536473 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-68bf65d8cf-lgp4z"] Oct 30 07:21:14.693941 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:14.693032 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:21:15.013053 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:15.013011 4951 patch_prober.go:29] interesting pod/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 07:21:15.013552 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:15.013077 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" podUID="5c1b682b-01eb-4717-9184-d81851ade4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 07:21:15.202596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:15.199659 4951 patch_prober.go:29] interesting pod/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 07:21:15.202596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:15.199719 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" podUID="9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 07:21:15.510609 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:15.508674 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-579b5d699b-7g76s" Oct 30 07:21:16.008083 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:16.008045 4951 patch_prober.go:29] interesting pod/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 07:21:16.008293 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:16.008115 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" podUID="5c1b682b-01eb-4717-9184-d81851ade4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 07:21:16.189926 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:16.189888 4951 patch_prober.go:29] interesting pod/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 07:21:16.190330 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:16.189946 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" podUID="9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 07:21:17.008957 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:17.008915 4951 patch_prober.go:29] interesting pod/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 07:21:17.009129 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:17.008989 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" podUID="5c1b682b-01eb-4717-9184-d81851ade4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 07:21:17.197864 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:17.195454 4951 patch_prober.go:29] interesting pod/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 07:21:17.197864 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:17.195530 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" podUID="9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 07:21:18.008379 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:18.008339 4951 patch_prober.go:29] interesting pod/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 07:21:18.008641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:18.008420 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" podUID="5c1b682b-01eb-4717-9184-d81851ade4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 07:21:18.190356 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:18.190313 4951 patch_prober.go:29] interesting pod/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 07:21:18.190597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:18.190427 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" podUID="9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 07:21:19.030968 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:19.030933 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:21:19.033308 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:19.033271 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:21:19.049536 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:19.049487 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:21:19.197016 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:19.196981 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:21:19.199471 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:19.199305 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:21:19.537318 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:19.537244 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-68bf65d8cf-lgp4z" podUID="5e4e9141-878d-424a-a658-87daf4272ddf" containerName="payload-pre-processing" containerID="cri-o://b423a31dfcf62a0367e1bf8b845219f6c60551a376e21cf9d8d4de5e80294222" gracePeriod=25 Oct 30 07:21:19.831149 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:19.831111 4951 generic.go:356] "Generic (PLEG): container finished" podID="5e4e9141-878d-424a-a658-87daf4272ddf" containerID="b423a31dfcf62a0367e1bf8b845219f6c60551a376e21cf9d8d4de5e80294222" exitCode=0 Oct 30 07:21:19.831328 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:19.831180 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-68bf65d8cf-lgp4z" event={"ID":"5e4e9141-878d-424a-a658-87daf4272ddf","Type":"ContainerDied","Data":"b423a31dfcf62a0367e1bf8b845219f6c60551a376e21cf9d8d4de5e80294222"} Oct 30 07:21:19.831328 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:19.831238 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-68bf65d8cf-lgp4z" event={"ID":"5e4e9141-878d-424a-a658-87daf4272ddf","Type":"ContainerDied","Data":"0091d406a4d3762097d8d08fc085b4cc07436dd45e5790b8b36b97adb1f8e9e6"} Oct 30 07:21:19.831328 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:19.831277 4951 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0091d406a4d3762097d8d08fc085b4cc07436dd45e5790b8b36b97adb1f8e9e6" Oct 30 07:21:19.835598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:19.835554 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-68bf65d8cf-lgp4z" Oct 30 07:21:19.902523 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:19.902479 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e4e9141-878d-424a-a658-87daf4272ddf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e4e9141-878d-424a-a658-87daf4272ddf-plugins-config-volume\") pod \"5e4e9141-878d-424a-a658-87daf4272ddf\" (UID: \"5e4e9141-878d-424a-a658-87daf4272ddf\") " Oct 30 07:21:19.902775 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:19.902702 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e4e9141-878d-424a-a658-87daf4272ddf-kube-api-access-z8x8w\" (UniqueName: \"kubernetes.io/projected/5e4e9141-878d-424a-a658-87daf4272ddf-kube-api-access-z8x8w\") pod \"5e4e9141-878d-424a-a658-87daf4272ddf\" (UID: \"5e4e9141-878d-424a-a658-87daf4272ddf\") " Oct 30 07:21:19.903005 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:19.902975 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e4e9141-878d-424a-a658-87daf4272ddf-plugins-config-volume" pod "5e4e9141-878d-424a-a658-87daf4272ddf" (UID: "5e4e9141-878d-424a-a658-87daf4272ddf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:21:19.903610 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:19.903547 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e4e9141-878d-424a-a658-87daf4272ddf-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:21:19.905815 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:19.905775 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e4e9141-878d-424a-a658-87daf4272ddf-kube-api-access-z8x8w" pod "5e4e9141-878d-424a-a658-87daf4272ddf" (UID: "5e4e9141-878d-424a-a658-87daf4272ddf"). InnerVolumeSpecName "kube-api-access-z8x8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:21:20.004855 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:20.004810 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8x8w\" (UniqueName: \"kubernetes.io/projected/5e4e9141-878d-424a-a658-87daf4272ddf-kube-api-access-z8x8w\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:21:20.839796 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:20.839763 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-68bf65d8cf-lgp4z" Oct 30 07:21:20.895461 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:20.895427 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-68bf65d8cf-lgp4z"] Oct 30 07:21:20.899971 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:20.899945 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-68bf65d8cf-lgp4z"] Oct 30 07:21:21.773089 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:21.773055 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e4e9141-878d-424a-a658-87daf4272ddf" path="/var/lib/kubelet/pods/5e4e9141-878d-424a-a658-87daf4272ddf/volumes" Oct 30 07:21:23.024514 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:21:23.024478 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:21:24.582141 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:24.582107 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-7cd9fbdcff-zn75b"] Oct 30 07:21:24.582996 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:24.582973 4951 cpu_manager.go:414] "containerMap: removing container" podUID="94280cc1-4ffe-4862-9cfc-1d61949679ed" containerName="istio-proxy" Oct 30 07:21:24.582996 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:24.582997 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94280cc1-4ffe-4862-9cfc-1d61949679ed" containerName="istio-proxy" Oct 30 07:21:24.583191 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:24.583026 4951 cpu_manager.go:414] "containerMap: removing container" podUID="91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e" containerName="payload-processing" Oct 30 07:21:24.583191 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:24.583034 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e" containerName="payload-processing" Oct 30 07:21:24.583191 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:24.583065 4951 cpu_manager.go:414] "containerMap: removing container" podUID="8889754c-55d5-4722-94dc-9579494fdd3e" containerName="payload-pre-processing" Oct 30 07:21:24.583191 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:24.583073 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8889754c-55d5-4722-94dc-9579494fdd3e" containerName="payload-pre-processing" Oct 30 07:21:24.583191 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:24.583089 4951 cpu_manager.go:414] "containerMap: removing container" podUID="5e4e9141-878d-424a-a658-87daf4272ddf" containerName="payload-pre-processing" Oct 30 07:21:24.583191 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:24.583098 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e4e9141-878d-424a-a658-87daf4272ddf" containerName="payload-pre-processing" Oct 30 07:21:24.583486 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:24.583283 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="91a4673f-4bc9-4d03-8fe2-72d8b9a51a4e" containerName="payload-processing" Oct 30 07:21:24.583486 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:24.583304 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="94280cc1-4ffe-4862-9cfc-1d61949679ed" containerName="istio-proxy" Oct 30 07:21:24.583486 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:24.583314 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="8889754c-55d5-4722-94dc-9579494fdd3e" containerName="payload-pre-processing" Oct 30 07:21:24.583486 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:24.583327 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e4e9141-878d-424a-a658-87daf4272ddf" containerName="payload-pre-processing" Oct 30 07:21:24.583486 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:24.583346 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="94280cc1-4ffe-4862-9cfc-1d61949679ed" containerName="istio-proxy" Oct 30 07:21:24.587606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:24.587539 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-7cd9fbdcff-zn75b" Oct 30 07:21:24.604171 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:24.604143 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-bdd496\"" Oct 30 07:21:24.623161 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:24.623124 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-7cd9fbdcff-zn75b"] Oct 30 07:21:24.623326 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:24.623308 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-bdd496-dockercfg-w97kq\"" Oct 30 07:21:24.778596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:24.758780 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d60f718d-86b8-4d75-a28a-7d7ad358cdb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bdd496-7cd9fbdcff-zn75b\" (UID: \"d60f718d-86b8-4d75-a28a-7d7ad358cdb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-7cd9fbdcff-zn75b" Oct 30 07:21:24.778596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:24.759050 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzv62\" (UniqueName: \"kubernetes.io/projected/d60f718d-86b8-4d75-a28a-7d7ad358cdb5-kube-api-access-mzv62\") pod \"payload-pre-processing-e2e-ipp-b-bdd496-7cd9fbdcff-zn75b\" (UID: \"d60f718d-86b8-4d75-a28a-7d7ad358cdb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-7cd9fbdcff-zn75b" Oct 30 07:21:24.895473 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:24.894945 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bdd496-55f7c4fcdd-v4qkn"] Oct 30 07:21:24.902968 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:24.901950 4951 cpu_manager.go:414] "containerMap: removing container" podUID="94280cc1-4ffe-4862-9cfc-1d61949679ed" containerName="istio-proxy" Oct 30 07:21:24.902968 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:24.901982 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94280cc1-4ffe-4862-9cfc-1d61949679ed" containerName="istio-proxy" Oct 30 07:21:24.920333 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:24.908121 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzv62\" (UniqueName: \"kubernetes.io/projected/d60f718d-86b8-4d75-a28a-7d7ad358cdb5-kube-api-access-mzv62\") pod \"payload-pre-processing-e2e-ipp-b-bdd496-7cd9fbdcff-zn75b\" (UID: \"d60f718d-86b8-4d75-a28a-7d7ad358cdb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-7cd9fbdcff-zn75b" Oct 30 07:21:24.920333 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:24.908274 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d60f718d-86b8-4d75-a28a-7d7ad358cdb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bdd496-7cd9fbdcff-zn75b\" (UID: \"d60f718d-86b8-4d75-a28a-7d7ad358cdb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-7cd9fbdcff-zn75b" Oct 30 07:21:24.920333 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:24.909504 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d60f718d-86b8-4d75-a28a-7d7ad358cdb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bdd496-7cd9fbdcff-zn75b\" (UID: \"d60f718d-86b8-4d75-a28a-7d7ad358cdb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-7cd9fbdcff-zn75b" Oct 30 07:21:24.920333 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:24.911997 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bdd496-55f7c4fcdd-v4qkn"] Oct 30 07:21:24.920333 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:24.912129 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-55f7c4fcdd-v4qkn" Oct 30 07:21:24.961272 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:24.961242 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzv62\" (UniqueName: \"kubernetes.io/projected/d60f718d-86b8-4d75-a28a-7d7ad358cdb5-kube-api-access-mzv62\") pod \"payload-pre-processing-e2e-ipp-b-bdd496-7cd9fbdcff-zn75b\" (UID: \"d60f718d-86b8-4d75-a28a-7d7ad358cdb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-7cd9fbdcff-zn75b" Oct 30 07:21:25.009251 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:25.009214 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hn6v\" (UniqueName: \"kubernetes.io/projected/20ccf7cf-223b-48aa-8845-af23df3d1c29-kube-api-access-2hn6v\") pod \"payload-processing-e2e-ipp-b-bdd496-55f7c4fcdd-v4qkn\" (UID: \"20ccf7cf-223b-48aa-8845-af23df3d1c29\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-55f7c4fcdd-v4qkn" Oct 30 07:21:25.009451 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:25.009309 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20ccf7cf-223b-48aa-8845-af23df3d1c29-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bdd496-55f7c4fcdd-v4qkn\" (UID: \"20ccf7cf-223b-48aa-8845-af23df3d1c29\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-55f7c4fcdd-v4qkn" Oct 30 07:21:25.110331 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:25.110291 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hn6v\" (UniqueName: \"kubernetes.io/projected/20ccf7cf-223b-48aa-8845-af23df3d1c29-kube-api-access-2hn6v\") pod \"payload-processing-e2e-ipp-b-bdd496-55f7c4fcdd-v4qkn\" (UID: \"20ccf7cf-223b-48aa-8845-af23df3d1c29\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-55f7c4fcdd-v4qkn" Oct 30 07:21:25.110513 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:25.110412 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20ccf7cf-223b-48aa-8845-af23df3d1c29-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bdd496-55f7c4fcdd-v4qkn\" (UID: \"20ccf7cf-223b-48aa-8845-af23df3d1c29\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-55f7c4fcdd-v4qkn" Oct 30 07:21:25.111168 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:25.111148 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20ccf7cf-223b-48aa-8845-af23df3d1c29-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bdd496-55f7c4fcdd-v4qkn\" (UID: \"20ccf7cf-223b-48aa-8845-af23df3d1c29\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-55f7c4fcdd-v4qkn" Oct 30 07:21:25.134271 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:25.134236 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hn6v\" (UniqueName: \"kubernetes.io/projected/20ccf7cf-223b-48aa-8845-af23df3d1c29-kube-api-access-2hn6v\") pod \"payload-processing-e2e-ipp-b-bdd496-55f7c4fcdd-v4qkn\" (UID: \"20ccf7cf-223b-48aa-8845-af23df3d1c29\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-55f7c4fcdd-v4qkn" Oct 30 07:21:25.205518 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:25.205424 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-7cd9fbdcff-zn75b" Oct 30 07:21:25.246939 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:25.246906 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-55f7c4fcdd-v4qkn" Oct 30 07:21:25.536379 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:25.536343 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-7cd9fbdcff-zn75b"] Oct 30 07:21:25.537006 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:21:25.536972 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd60f718d_86b8_4d75_a28a_7d7ad358cdb5.slice/crio-f2845de71c9af4eb7afe61cb5e18eb27c7e41e707324547569fffe98d86f3d38 WatchSource:0}: Error finding container f2845de71c9af4eb7afe61cb5e18eb27c7e41e707324547569fffe98d86f3d38: Status 404 returned error can't find the container with id f2845de71c9af4eb7afe61cb5e18eb27c7e41e707324547569fffe98d86f3d38 Oct 30 07:21:25.577874 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:25.577845 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bdd496-55f7c4fcdd-v4qkn"] Oct 30 07:21:25.582024 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:21:25.581993 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ccf7cf_223b_48aa_8845_af23df3d1c29.slice/crio-f042d3b360c6d0863507fc6acda43ac6907ca8b2625fe78b3665180d4f1787e5 WatchSource:0}: Error finding container f042d3b360c6d0863507fc6acda43ac6907ca8b2625fe78b3665180d4f1787e5: Status 404 returned error can't find the container with id f042d3b360c6d0863507fc6acda43ac6907ca8b2625fe78b3665180d4f1787e5 Oct 30 07:21:25.942447 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:25.942347 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-7cd9fbdcff-zn75b" event={"ID":"d60f718d-86b8-4d75-a28a-7d7ad358cdb5","Type":"ContainerStarted","Data":"3027d20743bdd8d9202d7748b4d497b763418b5e5dff48c9f5f02eb565c7e204"} Oct 30 07:21:25.942447 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:25.942416 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-7cd9fbdcff-zn75b" event={"ID":"d60f718d-86b8-4d75-a28a-7d7ad358cdb5","Type":"ContainerStarted","Data":"f2845de71c9af4eb7afe61cb5e18eb27c7e41e707324547569fffe98d86f3d38"} Oct 30 07:21:25.943023 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:25.942450 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-7cd9fbdcff-zn75b" Oct 30 07:21:25.944322 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:25.944293 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-55f7c4fcdd-v4qkn" event={"ID":"20ccf7cf-223b-48aa-8845-af23df3d1c29","Type":"ContainerStarted","Data":"aa46f4d423c666abdd53e8f378dd057d6da1bb47ab561f24ada9032f4fd7568d"} Oct 30 07:21:25.944462 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:25.944339 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-55f7c4fcdd-v4qkn" event={"ID":"20ccf7cf-223b-48aa-8845-af23df3d1c29","Type":"ContainerStarted","Data":"f042d3b360c6d0863507fc6acda43ac6907ca8b2625fe78b3665180d4f1787e5"} Oct 30 07:21:25.944462 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:25.944417 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-55f7c4fcdd-v4qkn" Oct 30 07:21:25.984043 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:25.983998 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-7cd9fbdcff-zn75b" podStartSLOduration=1.983980847 podStartE2EDuration="1.983980847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:21:25.982602889 +0000 UTC m=+1492.325650849" watchObservedRunningTime="2026-10-30 07:21:25.983980847 +0000 UTC m=+1492.327028812" Oct 30 07:21:26.031010 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:26.030953 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-55f7c4fcdd-v4qkn" podStartSLOduration=2.030936109 podStartE2EDuration="2.030936109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:21:26.026953518 +0000 UTC m=+1492.370001502" watchObservedRunningTime="2026-10-30 07:21:26.030936109 +0000 UTC m=+1492.373984099" Oct 30 07:21:29.472823 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:29.472786 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j"] Oct 30 07:21:29.477606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:29.477585 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j" Oct 30 07:21:29.506734 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:29.506704 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-fc546d-serving-cert\"" Oct 30 07:21:29.511081 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:29.511058 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j"] Oct 30 07:21:29.564590 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:29.564539 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgzbd\" (UniqueName: \"kubernetes.io/projected/837d2e3f-a5d4-4e78-aa65-3c277a685828-kube-api-access-rgzbd\") pod \"maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j\" (UID: \"837d2e3f-a5d4-4e78-aa65-3c277a685828\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j" Oct 30 07:21:29.564799 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:29.564653 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/837d2e3f-a5d4-4e78-aa65-3c277a685828-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j\" (UID: \"837d2e3f-a5d4-4e78-aa65-3c277a685828\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j" Oct 30 07:21:29.564799 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:29.564749 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/837d2e3f-a5d4-4e78-aa65-3c277a685828-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j\" (UID: \"837d2e3f-a5d4-4e78-aa65-3c277a685828\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j" Oct 30 07:21:29.564799 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:29.564785 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/837d2e3f-a5d4-4e78-aa65-3c277a685828-metrics-certs\") pod \"maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j\" (UID: \"837d2e3f-a5d4-4e78-aa65-3c277a685828\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j" Oct 30 07:21:29.666003 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:29.665964 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgzbd\" (UniqueName: \"kubernetes.io/projected/837d2e3f-a5d4-4e78-aa65-3c277a685828-kube-api-access-rgzbd\") pod \"maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j\" (UID: \"837d2e3f-a5d4-4e78-aa65-3c277a685828\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j" Oct 30 07:21:29.666157 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:29.666075 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/837d2e3f-a5d4-4e78-aa65-3c277a685828-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j\" (UID: \"837d2e3f-a5d4-4e78-aa65-3c277a685828\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j" Oct 30 07:21:29.666223 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:29.666169 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/837d2e3f-a5d4-4e78-aa65-3c277a685828-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j\" (UID: \"837d2e3f-a5d4-4e78-aa65-3c277a685828\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j" Oct 30 07:21:29.666223 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:29.666200 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/837d2e3f-a5d4-4e78-aa65-3c277a685828-metrics-certs\") pod \"maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j\" (UID: \"837d2e3f-a5d4-4e78-aa65-3c277a685828\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j" Oct 30 07:21:29.666946 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:29.666918 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/837d2e3f-a5d4-4e78-aa65-3c277a685828-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j\" (UID: \"837d2e3f-a5d4-4e78-aa65-3c277a685828\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j" Oct 30 07:21:29.669281 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:29.669254 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/837d2e3f-a5d4-4e78-aa65-3c277a685828-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j\" (UID: \"837d2e3f-a5d4-4e78-aa65-3c277a685828\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j" Oct 30 07:21:29.669397 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:29.669359 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/837d2e3f-a5d4-4e78-aa65-3c277a685828-metrics-certs\") pod \"maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j\" (UID: \"837d2e3f-a5d4-4e78-aa65-3c277a685828\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j" Oct 30 07:21:29.738652 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:29.723328 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgzbd\" (UniqueName: \"kubernetes.io/projected/837d2e3f-a5d4-4e78-aa65-3c277a685828-kube-api-access-rgzbd\") pod \"maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j\" (UID: \"837d2e3f-a5d4-4e78-aa65-3c277a685828\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j" Oct 30 07:21:29.820602 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:29.820006 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j" Oct 30 07:21:30.190637 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:30.190560 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j"] Oct 30 07:21:31.002559 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:31.002512 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j" event={"ID":"837d2e3f-a5d4-4e78-aa65-3c277a685828","Type":"ContainerStarted","Data":"8e078dcdb052a4fa710e48b3e9d7eb8dc71823e8a5fab45e3f45490c80df6418"} Oct 30 07:21:31.002933 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:31.002610 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j" event={"ID":"837d2e3f-a5d4-4e78-aa65-3c277a685828","Type":"ContainerStarted","Data":"9674891a75e9111a152d3ea8b2ae4eaf096f68548720894bef8d6241167c6c31"} Oct 30 07:21:31.002933 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:31.002640 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j" Oct 30 07:21:31.048597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:31.048032 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j" podStartSLOduration=1.67507579 podStartE2EDuration="2.048011841s" totalImagesPullingTime="372.936051ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:21:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:21:31.045009781 +0000 UTC m=+1497.388057768" watchObservedRunningTime="2026-10-30 07:21:31.048011841 +0000 UTC m=+1497.391059807" Oct 30 07:21:34.186788 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:34.186761 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/3.log" Oct 30 07:21:34.188865 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:34.188843 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/3.log" Oct 30 07:21:34.191932 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:34.191911 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 07:21:34.194204 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:34.194187 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 07:21:34.719027 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:34.718996 4951 scope.go:122] "RemoveContainer" containerID="2e5d0e999ed61144f7d174347904f6336a028f6c633f867369664e0fea660c8c" Oct 30 07:21:35.080606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:35.080095 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efce7b-86c7d47c8d-8tvds"] Oct 30 07:21:35.095600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:35.092745 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efce7b-86c7d47c8d-8tvds"] Oct 30 07:21:35.095600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:35.092872 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-86c7d47c8d-8tvds" Oct 30 07:21:35.123628 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:35.121645 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-efce7b\"" Oct 30 07:21:35.123628 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:35.121756 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-efce7b-dockercfg-wf2q6\"" Oct 30 07:21:35.138423 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:35.138106 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd6mx\" (UniqueName: \"kubernetes.io/projected/f4d71ae8-3979-48f6-ab97-612b599dd6c4-kube-api-access-qd6mx\") pod \"payload-processing-e2e-ipp-b-efce7b-86c7d47c8d-8tvds\" (UID: \"f4d71ae8-3979-48f6-ab97-612b599dd6c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-86c7d47c8d-8tvds" Oct 30 07:21:35.138423 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:35.138177 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4d71ae8-3979-48f6-ab97-612b599dd6c4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-efce7b-86c7d47c8d-8tvds\" (UID: \"f4d71ae8-3979-48f6-ab97-612b599dd6c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-86c7d47c8d-8tvds" Oct 30 07:21:35.239564 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:35.239520 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd6mx\" (UniqueName: \"kubernetes.io/projected/f4d71ae8-3979-48f6-ab97-612b599dd6c4-kube-api-access-qd6mx\") pod \"payload-processing-e2e-ipp-b-efce7b-86c7d47c8d-8tvds\" (UID: \"f4d71ae8-3979-48f6-ab97-612b599dd6c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-86c7d47c8d-8tvds" Oct 30 07:21:35.240008 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:35.239648 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4d71ae8-3979-48f6-ab97-612b599dd6c4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-efce7b-86c7d47c8d-8tvds\" (UID: \"f4d71ae8-3979-48f6-ab97-612b599dd6c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-86c7d47c8d-8tvds" Oct 30 07:21:35.240560 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:35.240534 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4d71ae8-3979-48f6-ab97-612b599dd6c4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-efce7b-86c7d47c8d-8tvds\" (UID: \"f4d71ae8-3979-48f6-ab97-612b599dd6c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-86c7d47c8d-8tvds" Oct 30 07:21:35.263847 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:35.263818 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd6mx\" (UniqueName: \"kubernetes.io/projected/f4d71ae8-3979-48f6-ab97-612b599dd6c4-kube-api-access-qd6mx\") pod \"payload-processing-e2e-ipp-b-efce7b-86c7d47c8d-8tvds\" (UID: \"f4d71ae8-3979-48f6-ab97-612b599dd6c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-86c7d47c8d-8tvds" Oct 30 07:21:35.283833 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:35.283805 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-17bd11-57864cfc76-zw5qj"] Oct 30 07:21:35.320310 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:35.320282 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-59c44dfbbc-4c96d"] Oct 30 07:21:35.427129 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:35.421703 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-86c7d47c8d-8tvds" Oct 30 07:21:35.845857 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:35.845781 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efce7b-86c7d47c8d-8tvds"] Oct 30 07:21:36.164726 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:36.154037 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-86c7d47c8d-8tvds" event={"ID":"f4d71ae8-3979-48f6-ab97-612b599dd6c4","Type":"ContainerStarted","Data":"0f93525973df6f1ceea43366b36a593d4eba4287dcde1cfd73989d7d78ac7454"} Oct 30 07:21:36.164726 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:36.154174 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-86c7d47c8d-8tvds" event={"ID":"f4d71ae8-3979-48f6-ab97-612b599dd6c4","Type":"ContainerStarted","Data":"e5002a983d251ccb9985f057121d1defd286366e5a01b7f0ed51ccbfd79f568b"} Oct 30 07:21:36.164726 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:36.155410 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-86c7d47c8d-8tvds" Oct 30 07:21:36.256595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:36.244707 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-86c7d47c8d-8tvds" podStartSLOduration=1.24468288 podStartE2EDuration="1.24468288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:21:36.241401713 +0000 UTC m=+1502.584449696" watchObservedRunningTime="2026-10-30 07:21:36.24468288 +0000 UTC m=+1502.587730847" Oct 30 07:21:36.957401 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:36.957367 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-7cd9fbdcff-zn75b" Oct 30 07:21:36.957606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:36.957456 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-55f7c4fcdd-v4qkn" Oct 30 07:21:37.017609 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:37.017540 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j" Oct 30 07:21:40.189835 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:40.189795 4951 generic.go:356] "Generic (PLEG): container finished" podID="b0938f71-f1d8-4342-8d0a-7aed59adef39" containerID="74378f4a114b21f52a0d256fd583040199effa222ac25fd48a58b1e50a660be2" exitCode=0 Oct 30 07:21:40.190292 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:40.189865 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" event={"ID":"b0938f71-f1d8-4342-8d0a-7aed59adef39","Type":"ContainerDied","Data":"74378f4a114b21f52a0d256fd583040199effa222ac25fd48a58b1e50a660be2"} Oct 30 07:21:40.190348 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:40.190299 4951 scope.go:122] "RemoveContainer" containerID="74378f4a114b21f52a0d256fd583040199effa222ac25fd48a58b1e50a660be2" Oct 30 07:21:40.191911 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:40.191884 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:21:40.191991 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:40.191954 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:21:40.191991 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:40.191987 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:21:40.285019 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:40.284987 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-57864cfc76-zw5qj" podUID="7e588a7d-d969-4313-aa99-54cd1776829c" containerName="payload-processing" containerID="cri-o://78d873a6b22881648f2054ab1865dcd83c557f7d4938d2fdcc6f295b74607387" gracePeriod=25 Oct 30 07:21:40.322732 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:40.322401 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-59c44dfbbc-4c96d" podUID="b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e" containerName="payload-pre-processing" containerID="cri-o://1c1dfc39d023b9a7b3f8d887ba72222881d768f6945894522c97b4a9cc87bfed" gracePeriod=25 Oct 30 07:21:40.673426 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:40.673386 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-57864cfc76-zw5qj" Oct 30 07:21:40.732859 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:40.732789 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-59c44dfbbc-4c96d" Oct 30 07:21:40.785176 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:40.785129 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e-kube-api-access-5pghm\" (UniqueName: \"kubernetes.io/projected/b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e-kube-api-access-5pghm\") pod \"b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e\" (UID: \"b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e\") " Oct 30 07:21:40.785632 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:40.785606 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e588a7d-d969-4313-aa99-54cd1776829c-kube-api-access-4xjtr\" (UniqueName: \"kubernetes.io/projected/7e588a7d-d969-4313-aa99-54cd1776829c-kube-api-access-4xjtr\") pod \"7e588a7d-d969-4313-aa99-54cd1776829c\" (UID: \"7e588a7d-d969-4313-aa99-54cd1776829c\") " Oct 30 07:21:40.785893 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:40.785876 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e-plugins-config-volume\") pod \"b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e\" (UID: \"b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e\") " Oct 30 07:21:40.786537 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:40.786397 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e-plugins-config-volume" pod "b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e" (UID: "b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:21:40.798701 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:40.788879 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e588a7d-d969-4313-aa99-54cd1776829c-plugins-config-volume" pod "7e588a7d-d969-4313-aa99-54cd1776829c" (UID: "7e588a7d-d969-4313-aa99-54cd1776829c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:21:40.798879 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:40.786469 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e588a7d-d969-4313-aa99-54cd1776829c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e588a7d-d969-4313-aa99-54cd1776829c-plugins-config-volume\") pod \"7e588a7d-d969-4313-aa99-54cd1776829c\" (UID: \"7e588a7d-d969-4313-aa99-54cd1776829c\") " Oct 30 07:21:40.800261 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:40.799859 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:21:40.805749 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:40.800344 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e588a7d-d969-4313-aa99-54cd1776829c-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:21:40.805749 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:40.800450 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e-kube-api-access-5pghm" pod "b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e" (UID: "b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e"). InnerVolumeSpecName "kube-api-access-5pghm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:21:40.805749 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:40.803821 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e588a7d-d969-4313-aa99-54cd1776829c-kube-api-access-4xjtr" pod "7e588a7d-d969-4313-aa99-54cd1776829c" (UID: "7e588a7d-d969-4313-aa99-54cd1776829c"). InnerVolumeSpecName "kube-api-access-4xjtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:21:40.901998 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:40.901966 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pghm\" (UniqueName: \"kubernetes.io/projected/b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e-kube-api-access-5pghm\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:21:40.901998 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:40.901994 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xjtr\" (UniqueName: \"kubernetes.io/projected/7e588a7d-d969-4313-aa99-54cd1776829c-kube-api-access-4xjtr\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:21:41.197814 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:41.197779 4951 generic.go:356] "Generic (PLEG): container finished" podID="b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e" containerID="1c1dfc39d023b9a7b3f8d887ba72222881d768f6945894522c97b4a9cc87bfed" exitCode=0 Oct 30 07:21:41.198319 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:41.197891 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-59c44dfbbc-4c96d" Oct 30 07:21:41.198319 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:41.197938 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-59c44dfbbc-4c96d" event={"ID":"b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e","Type":"ContainerDied","Data":"1c1dfc39d023b9a7b3f8d887ba72222881d768f6945894522c97b4a9cc87bfed"} Oct 30 07:21:41.198319 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:41.198010 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-59c44dfbbc-4c96d" event={"ID":"b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e","Type":"ContainerDied","Data":"6b00f6ccf9c7f8d9e6cccc52119b545d6a7c9ca7244f1d56a831eb6d5391ae6d"} Oct 30 07:21:41.198319 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:41.198042 4951 scope.go:122] "RemoveContainer" containerID="1c1dfc39d023b9a7b3f8d887ba72222881d768f6945894522c97b4a9cc87bfed" Oct 30 07:21:41.199474 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:41.199376 4951 generic.go:356] "Generic (PLEG): container finished" podID="7e588a7d-d969-4313-aa99-54cd1776829c" containerID="78d873a6b22881648f2054ab1865dcd83c557f7d4938d2fdcc6f295b74607387" exitCode=0 Oct 30 07:21:41.199588 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:41.199480 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-57864cfc76-zw5qj" Oct 30 07:21:41.199588 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:41.199498 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-57864cfc76-zw5qj" event={"ID":"7e588a7d-d969-4313-aa99-54cd1776829c","Type":"ContainerDied","Data":"78d873a6b22881648f2054ab1865dcd83c557f7d4938d2fdcc6f295b74607387"} Oct 30 07:21:41.199588 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:41.199547 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-57864cfc76-zw5qj" event={"ID":"7e588a7d-d969-4313-aa99-54cd1776829c","Type":"ContainerDied","Data":"e8020700c918a939319bafe18bb668b821ffc1112c2d24368efa11294f7e0966"} Oct 30 07:21:41.202661 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:41.202631 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" event={"ID":"b0938f71-f1d8-4342-8d0a-7aed59adef39","Type":"ContainerStarted","Data":"c30a4cba4d5a0410ce429bd419852e14580de8fe34bc5b07bd2e1a149718e385"} Oct 30 07:21:41.211843 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:41.211820 4951 scope.go:122] "RemoveContainer" containerID="1c1dfc39d023b9a7b3f8d887ba72222881d768f6945894522c97b4a9cc87bfed" Oct 30 07:21:41.212222 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:21:41.212193 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c1dfc39d023b9a7b3f8d887ba72222881d768f6945894522c97b4a9cc87bfed\": container with ID starting with 1c1dfc39d023b9a7b3f8d887ba72222881d768f6945894522c97b4a9cc87bfed not found: ID does not exist" containerID="1c1dfc39d023b9a7b3f8d887ba72222881d768f6945894522c97b4a9cc87bfed" Oct 30 07:21:41.212334 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:41.212235 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c1dfc39d023b9a7b3f8d887ba72222881d768f6945894522c97b4a9cc87bfed"} err="failed to get container status \"1c1dfc39d023b9a7b3f8d887ba72222881d768f6945894522c97b4a9cc87bfed\": rpc error: code = NotFound desc = could not find container \"1c1dfc39d023b9a7b3f8d887ba72222881d768f6945894522c97b4a9cc87bfed\": container with ID starting with 1c1dfc39d023b9a7b3f8d887ba72222881d768f6945894522c97b4a9cc87bfed not found: ID does not exist" Oct 30 07:21:41.212334 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:41.212265 4951 scope.go:122] "RemoveContainer" containerID="78d873a6b22881648f2054ab1865dcd83c557f7d4938d2fdcc6f295b74607387" Oct 30 07:21:41.249797 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:41.244639 4951 scope.go:122] "RemoveContainer" containerID="78d873a6b22881648f2054ab1865dcd83c557f7d4938d2fdcc6f295b74607387" Oct 30 07:21:41.249797 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:21:41.246666 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78d873a6b22881648f2054ab1865dcd83c557f7d4938d2fdcc6f295b74607387\": container with ID starting with 78d873a6b22881648f2054ab1865dcd83c557f7d4938d2fdcc6f295b74607387 not found: ID does not exist" containerID="78d873a6b22881648f2054ab1865dcd83c557f7d4938d2fdcc6f295b74607387" Oct 30 07:21:41.249797 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:41.246729 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78d873a6b22881648f2054ab1865dcd83c557f7d4938d2fdcc6f295b74607387"} err="failed to get container status \"78d873a6b22881648f2054ab1865dcd83c557f7d4938d2fdcc6f295b74607387\": rpc error: code = NotFound desc = could not find container \"78d873a6b22881648f2054ab1865dcd83c557f7d4938d2fdcc6f295b74607387\": container with ID starting with 78d873a6b22881648f2054ab1865dcd83c557f7d4938d2fdcc6f295b74607387 not found: ID does not exist" Oct 30 07:21:41.263273 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:41.263229 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:21:41.264183 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:41.264157 4951 patch_prober.go:29] interesting pod/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 07:21:41.264320 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:41.264209 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" podUID="b0938f71-f1d8-4342-8d0a-7aed59adef39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 07:21:41.269735 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:41.269708 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-59c44dfbbc-4c96d"] Oct 30 07:21:41.281171 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:41.281146 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-59c44dfbbc-4c96d"] Oct 30 07:21:41.362237 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:41.362203 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-17bd11-57864cfc76-zw5qj"] Oct 30 07:21:41.368083 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:41.368055 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-17bd11-57864cfc76-zw5qj"] Oct 30 07:21:41.773141 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:41.773103 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e588a7d-d969-4313-aa99-54cd1776829c" path="/var/lib/kubelet/pods/7e588a7d-d969-4313-aa99-54cd1776829c/volumes" Oct 30 07:21:41.773714 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:41.773693 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e" path="/var/lib/kubelet/pods/b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e/volumes" Oct 30 07:21:42.263452 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:42.263418 4951 patch_prober.go:29] interesting pod/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 07:21:42.263986 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:42.263473 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" podUID="b0938f71-f1d8-4342-8d0a-7aed59adef39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 07:21:42.488057 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:42.488022 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:21:43.263328 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:43.263289 4951 patch_prober.go:29] interesting pod/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 07:21:43.263543 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:43.263344 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" podUID="b0938f71-f1d8-4342-8d0a-7aed59adef39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 07:21:44.267592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:44.267015 4951 patch_prober.go:29] interesting pod/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 07:21:44.267592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:44.267082 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" podUID="b0938f71-f1d8-4342-8d0a-7aed59adef39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 07:21:45.263442 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:45.263402 4951 patch_prober.go:29] interesting pod/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 07:21:45.263700 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:45.263454 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" podUID="b0938f71-f1d8-4342-8d0a-7aed59adef39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 07:21:46.263391 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:46.263345 4951 patch_prober.go:29] interesting pod/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 07:21:46.264054 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:46.263429 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" podUID="b0938f71-f1d8-4342-8d0a-7aed59adef39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 07:21:47.269537 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:47.269497 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:21:47.271007 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:47.270980 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:21:48.175629 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:21:48.175600 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-86c7d47c8d-8tvds" Oct 30 07:21:53.088719 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:21:53.088674 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:22:00.616618 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:00.605407 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-58bf9bd7f9-7m5cm"] Oct 30 07:22:00.620187 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:00.620160 4951 cpu_manager.go:414] "containerMap: removing container" podUID="b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e" containerName="payload-pre-processing" Oct 30 07:22:00.624684 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:00.624640 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e" containerName="payload-pre-processing" Oct 30 07:22:00.625028 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:00.624993 4951 cpu_manager.go:414] "containerMap: removing container" podUID="7e588a7d-d969-4313-aa99-54cd1776829c" containerName="payload-processing" Oct 30 07:22:00.625157 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:00.625143 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e588a7d-d969-4313-aa99-54cd1776829c" containerName="payload-processing" Oct 30 07:22:00.625606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:00.625548 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0a415ba-5c75-4ae9-bbf2-60c68eb8a53e" containerName="payload-pre-processing" Oct 30 07:22:00.625788 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:00.625772 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e588a7d-d969-4313-aa99-54cd1776829c" containerName="payload-processing" Oct 30 07:22:00.648273 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:00.648243 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-58bf9bd7f9-7m5cm" Oct 30 07:22:00.715628 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:00.702522 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-17bd11\"" Oct 30 07:22:00.776859 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:00.769937 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-17bd11-dockercfg-kt8xt\"" Oct 30 07:22:00.778779 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:00.778751 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-58bf9bd7f9-7m5cm"] Oct 30 07:22:00.783953 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:00.782502 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-17bd11-8c45f755d-2tr2x"] Oct 30 07:22:00.808510 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:00.808479 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-8c45f755d-2tr2x" Oct 30 07:22:00.821513 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:00.821469 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-17bd11-8c45f755d-2tr2x"] Oct 30 07:22:00.848675 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:00.847809 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e60656-b3e9-4438-b73e-91e2632e452f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-17bd11-58bf9bd7f9-7m5cm\" (UID: \"41e60656-b3e9-4438-b73e-91e2632e452f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-58bf9bd7f9-7m5cm" Oct 30 07:22:00.848675 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:00.847944 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnl4d\" (UniqueName: \"kubernetes.io/projected/41e60656-b3e9-4438-b73e-91e2632e452f-kube-api-access-qnl4d\") pod \"payload-pre-processing-e2e-ipp-a-17bd11-58bf9bd7f9-7m5cm\" (UID: \"41e60656-b3e9-4438-b73e-91e2632e452f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-58bf9bd7f9-7m5cm" Oct 30 07:22:00.952163 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:00.950056 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e60656-b3e9-4438-b73e-91e2632e452f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-17bd11-58bf9bd7f9-7m5cm\" (UID: \"41e60656-b3e9-4438-b73e-91e2632e452f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-58bf9bd7f9-7m5cm" Oct 30 07:22:00.952163 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:00.950133 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnn9d\" (UniqueName: \"kubernetes.io/projected/a133e6d0-be93-4023-9831-7dacca337709-kube-api-access-wnn9d\") pod \"payload-processing-e2e-ipp-a-17bd11-8c45f755d-2tr2x\" (UID: \"a133e6d0-be93-4023-9831-7dacca337709\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-8c45f755d-2tr2x" Oct 30 07:22:00.952163 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:00.950231 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnl4d\" (UniqueName: \"kubernetes.io/projected/41e60656-b3e9-4438-b73e-91e2632e452f-kube-api-access-qnl4d\") pod \"payload-pre-processing-e2e-ipp-a-17bd11-58bf9bd7f9-7m5cm\" (UID: \"41e60656-b3e9-4438-b73e-91e2632e452f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-58bf9bd7f9-7m5cm" Oct 30 07:22:00.952163 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:00.950314 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a133e6d0-be93-4023-9831-7dacca337709-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-17bd11-8c45f755d-2tr2x\" (UID: \"a133e6d0-be93-4023-9831-7dacca337709\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-8c45f755d-2tr2x" Oct 30 07:22:00.953365 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:00.953335 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e60656-b3e9-4438-b73e-91e2632e452f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-17bd11-58bf9bd7f9-7m5cm\" (UID: \"41e60656-b3e9-4438-b73e-91e2632e452f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-58bf9bd7f9-7m5cm" Oct 30 07:22:00.975839 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:00.975800 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnl4d\" (UniqueName: \"kubernetes.io/projected/41e60656-b3e9-4438-b73e-91e2632e452f-kube-api-access-qnl4d\") pod \"payload-pre-processing-e2e-ipp-a-17bd11-58bf9bd7f9-7m5cm\" (UID: \"41e60656-b3e9-4438-b73e-91e2632e452f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-58bf9bd7f9-7m5cm" Oct 30 07:22:01.052103 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:01.052047 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnn9d\" (UniqueName: \"kubernetes.io/projected/a133e6d0-be93-4023-9831-7dacca337709-kube-api-access-wnn9d\") pod \"payload-processing-e2e-ipp-a-17bd11-8c45f755d-2tr2x\" (UID: \"a133e6d0-be93-4023-9831-7dacca337709\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-8c45f755d-2tr2x" Oct 30 07:22:01.052311 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:01.052189 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a133e6d0-be93-4023-9831-7dacca337709-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-17bd11-8c45f755d-2tr2x\" (UID: \"a133e6d0-be93-4023-9831-7dacca337709\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-8c45f755d-2tr2x" Oct 30 07:22:01.053033 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:01.052997 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a133e6d0-be93-4023-9831-7dacca337709-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-17bd11-8c45f755d-2tr2x\" (UID: \"a133e6d0-be93-4023-9831-7dacca337709\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-8c45f755d-2tr2x" Oct 30 07:22:01.075974 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:01.075943 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnn9d\" (UniqueName: \"kubernetes.io/projected/a133e6d0-be93-4023-9831-7dacca337709-kube-api-access-wnn9d\") pod \"payload-processing-e2e-ipp-a-17bd11-8c45f755d-2tr2x\" (UID: \"a133e6d0-be93-4023-9831-7dacca337709\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-8c45f755d-2tr2x" Oct 30 07:22:01.093514 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:01.093478 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-58bf9bd7f9-7m5cm" Oct 30 07:22:01.135036 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:01.135001 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-8c45f755d-2tr2x" Oct 30 07:22:01.484657 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:01.483344 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-58bf9bd7f9-7m5cm"] Oct 30 07:22:01.497613 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:01.497512 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-58bf9bd7f9-7m5cm" event={"ID":"41e60656-b3e9-4438-b73e-91e2632e452f","Type":"ContainerStarted","Data":"d7a467277ccc782334418296c44ee16e2e346d6ca1bec3d50b530fd70dfec1b1"} Oct 30 07:22:01.544135 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:01.543893 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-17bd11-8c45f755d-2tr2x"] Oct 30 07:22:02.508224 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:02.508184 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-58bf9bd7f9-7m5cm" event={"ID":"41e60656-b3e9-4438-b73e-91e2632e452f","Type":"ContainerStarted","Data":"44806be4f52b4ef13df780a24a249c52d8ab060c919c26ae359e10c2533aeb53"} Oct 30 07:22:02.508736 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:02.508345 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-58bf9bd7f9-7m5cm" Oct 30 07:22:02.510026 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:02.509998 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-8c45f755d-2tr2x" event={"ID":"a133e6d0-be93-4023-9831-7dacca337709","Type":"ContainerStarted","Data":"7473c8c2a202f23e616dec718ac0d09fdd5983efaa5d706051ad8f72d9e0c97c"} Oct 30 07:22:02.510147 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:02.510053 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-8c45f755d-2tr2x" event={"ID":"a133e6d0-be93-4023-9831-7dacca337709","Type":"ContainerStarted","Data":"2a763b059e4b9bc4e0cf220e63d83f9f835395bef5cc3d2aa25cccd5f2c2b481"} Oct 30 07:22:02.510147 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:02.510083 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-8c45f755d-2tr2x" Oct 30 07:22:02.550771 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:02.550352 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-58bf9bd7f9-7m5cm" podStartSLOduration=2.5503325 podStartE2EDuration="2.5503325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:22:02.548548099 +0000 UTC m=+1528.891596067" watchObservedRunningTime="2026-10-30 07:22:02.5503325 +0000 UTC m=+1528.893380467" Oct 30 07:22:02.593909 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:02.593848 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-8c45f755d-2tr2x" podStartSLOduration=2.593830033 podStartE2EDuration="2.593830033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:22:02.589397991 +0000 UTC m=+1528.932445960" watchObservedRunningTime="2026-10-30 07:22:02.593830033 +0000 UTC m=+1528.936877999" Oct 30 07:22:03.518631 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:03.518563 4951 generic.go:356] "Generic (PLEG): container finished" podID="5ed0bc5e-986d-4699-9513-053f7fe24b65" containerID="800de26dd48119553231546144674293d0dbf81bdbd6ecf9c935135284d2aabb" exitCode=0 Oct 30 07:22:03.519128 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:03.518601 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" event={"ID":"5ed0bc5e-986d-4699-9513-053f7fe24b65","Type":"ContainerDied","Data":"800de26dd48119553231546144674293d0dbf81bdbd6ecf9c935135284d2aabb"} Oct 30 07:22:03.519360 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:03.519344 4951 scope.go:122] "RemoveContainer" containerID="800de26dd48119553231546144674293d0dbf81bdbd6ecf9c935135284d2aabb" Oct 30 07:22:03.520813 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:03.520781 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:22:03.520936 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:03.520867 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:22:03.520936 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:03.520913 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:22:04.559030 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:04.558991 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" event={"ID":"5ed0bc5e-986d-4699-9513-053f7fe24b65","Type":"ContainerStarted","Data":"d635ac3b796d2edd565ddb1e5fa423d6e7e6ce478f15bb5201d4c4a406dbece5"} Oct 30 07:22:04.805427 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:04.805395 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:22:04.806865 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:04.806840 4951 patch_prober.go:29] interesting pod/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 07:22:04.806996 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:04.806889 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" podUID="5ed0bc5e-986d-4699-9513-053f7fe24b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 07:22:05.494993 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:05.494957 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:22:05.805954 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:05.805917 4951 patch_prober.go:29] interesting pod/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 07:22:05.806432 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:05.805963 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" podUID="5ed0bc5e-986d-4699-9513-053f7fe24b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 07:22:06.806238 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:06.806198 4951 patch_prober.go:29] interesting pod/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 07:22:06.806759 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:06.806278 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" podUID="5ed0bc5e-986d-4699-9513-053f7fe24b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 07:22:07.806222 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:07.806189 4951 patch_prober.go:29] interesting pod/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 07:22:07.806402 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:07.806241 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" podUID="5ed0bc5e-986d-4699-9513-053f7fe24b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 07:22:08.805481 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:08.805447 4951 patch_prober.go:29] interesting pod/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 07:22:08.805662 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:08.805515 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" podUID="5ed0bc5e-986d-4699-9513-053f7fe24b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 07:22:09.805440 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:09.805404 4951 patch_prober.go:29] interesting pod/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 07:22:09.806025 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:09.805466 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" podUID="5ed0bc5e-986d-4699-9513-053f7fe24b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 07:22:10.182897 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.182820 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4"] Oct 30 07:22:10.183055 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.183030 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4" podUID="e4f5cb00-0e55-4cc7-8599-1b6d5d94de02" containerName="maas-api" containerID="cri-o://ac131475081d2d18791ad863c39547f311df749c0fe11cf8f6acea2e5a3f69a3" gracePeriod=30 Oct 30 07:22:10.360469 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.360431 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-6bbb96bf6d-wwc5q"] Oct 30 07:22:10.360961 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.360713 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-6bbb96bf6d-wwc5q" podUID="d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450" containerName="payload-processing" containerID="cri-o://1fb10342388d7c014b375457cc3f79d06ddf3b48b57ea64a8af53f0211b6bb76" gracePeriod=30 Oct 30 07:22:10.383453 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.381062 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-794fc67c4c-z2hjz"] Oct 30 07:22:10.383453 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.381302 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-794fc67c4c-z2hjz" podUID="4a995882-9616-44c9-8f3c-2d6615d7bbde" containerName="payload-pre-processing" containerID="cri-o://cd75c1b7ea8ceef193e78de57f949a9d189c471cb866fa82ccafc4fdf3c86d70" gracePeriod=30 Oct 30 07:22:10.565651 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.565619 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4" Oct 30 07:22:10.634631 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.621648 4951 generic.go:356] "Generic (PLEG): container finished" podID="d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450" containerID="1fb10342388d7c014b375457cc3f79d06ddf3b48b57ea64a8af53f0211b6bb76" exitCode=0 Oct 30 07:22:10.634631 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.621767 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-6bbb96bf6d-wwc5q" event={"ID":"d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450","Type":"ContainerDied","Data":"1fb10342388d7c014b375457cc3f79d06ddf3b48b57ea64a8af53f0211b6bb76"} Oct 30 07:22:10.634631 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.624974 4951 generic.go:356] "Generic (PLEG): container finished" podID="4a995882-9616-44c9-8f3c-2d6615d7bbde" containerID="cd75c1b7ea8ceef193e78de57f949a9d189c471cb866fa82ccafc4fdf3c86d70" exitCode=0 Oct 30 07:22:10.634631 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.625101 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-794fc67c4c-z2hjz" event={"ID":"4a995882-9616-44c9-8f3c-2d6615d7bbde","Type":"ContainerDied","Data":"cd75c1b7ea8ceef193e78de57f949a9d189c471cb866fa82ccafc4fdf3c86d70"} Oct 30 07:22:10.636355 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.636316 4951 generic.go:356] "Generic (PLEG): container finished" podID="e4f5cb00-0e55-4cc7-8599-1b6d5d94de02" containerID="ac131475081d2d18791ad863c39547f311df749c0fe11cf8f6acea2e5a3f69a3" exitCode=0 Oct 30 07:22:10.636525 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.636404 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4" event={"ID":"e4f5cb00-0e55-4cc7-8599-1b6d5d94de02","Type":"ContainerDied","Data":"ac131475081d2d18791ad863c39547f311df749c0fe11cf8f6acea2e5a3f69a3"} Oct 30 07:22:10.636525 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.636455 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4" event={"ID":"e4f5cb00-0e55-4cc7-8599-1b6d5d94de02","Type":"ContainerDied","Data":"c7fa307ac884221ea3feb0029c28b711c2fba2a1f74c4c1bf8ae7f6a5f06571c"} Oct 30 07:22:10.636525 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.636485 4951 scope.go:122] "RemoveContainer" containerID="ac131475081d2d18791ad863c39547f311df749c0fe11cf8f6acea2e5a3f69a3" Oct 30 07:22:10.636747 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.636714 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4" Oct 30 07:22:10.662029 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.659804 4951 scope.go:122] "RemoveContainer" containerID="ac131475081d2d18791ad863c39547f311df749c0fe11cf8f6acea2e5a3f69a3" Oct 30 07:22:10.662706 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:22:10.662666 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac131475081d2d18791ad863c39547f311df749c0fe11cf8f6acea2e5a3f69a3\": container with ID starting with ac131475081d2d18791ad863c39547f311df749c0fe11cf8f6acea2e5a3f69a3 not found: ID does not exist" containerID="ac131475081d2d18791ad863c39547f311df749c0fe11cf8f6acea2e5a3f69a3" Oct 30 07:22:10.662831 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.662705 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac131475081d2d18791ad863c39547f311df749c0fe11cf8f6acea2e5a3f69a3"} err="failed to get container status \"ac131475081d2d18791ad863c39547f311df749c0fe11cf8f6acea2e5a3f69a3\": rpc error: code = NotFound desc = could not find container \"ac131475081d2d18791ad863c39547f311df749c0fe11cf8f6acea2e5a3f69a3\": container with ID starting with ac131475081d2d18791ad863c39547f311df749c0fe11cf8f6acea2e5a3f69a3 not found: ID does not exist" Oct 30 07:22:10.681320 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.681288 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-6bbb96bf6d-wwc5q" Oct 30 07:22:10.704711 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.704672 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4f5cb00-0e55-4cc7-8599-1b6d5d94de02-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4f5cb00-0e55-4cc7-8599-1b6d5d94de02-maas-api-tls\") pod \"e4f5cb00-0e55-4cc7-8599-1b6d5d94de02\" (UID: \"e4f5cb00-0e55-4cc7-8599-1b6d5d94de02\") " Oct 30 07:22:10.704896 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.704820 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4f5cb00-0e55-4cc7-8599-1b6d5d94de02-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4f5cb00-0e55-4cc7-8599-1b6d5d94de02-metrics-certs\") pod \"e4f5cb00-0e55-4cc7-8599-1b6d5d94de02\" (UID: \"e4f5cb00-0e55-4cc7-8599-1b6d5d94de02\") " Oct 30 07:22:10.704896 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.704864 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4f5cb00-0e55-4cc7-8599-1b6d5d94de02-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4f5cb00-0e55-4cc7-8599-1b6d5d94de02-openshift-service-ca\") pod \"e4f5cb00-0e55-4cc7-8599-1b6d5d94de02\" (UID: \"e4f5cb00-0e55-4cc7-8599-1b6d5d94de02\") " Oct 30 07:22:10.705037 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.704944 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4f5cb00-0e55-4cc7-8599-1b6d5d94de02-kube-api-access-8pmct\" (UniqueName: \"kubernetes.io/projected/e4f5cb00-0e55-4cc7-8599-1b6d5d94de02-kube-api-access-8pmct\") pod \"e4f5cb00-0e55-4cc7-8599-1b6d5d94de02\" (UID: \"e4f5cb00-0e55-4cc7-8599-1b6d5d94de02\") " Oct 30 07:22:10.705899 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.705866 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4f5cb00-0e55-4cc7-8599-1b6d5d94de02-openshift-service-ca" pod "e4f5cb00-0e55-4cc7-8599-1b6d5d94de02" (UID: "e4f5cb00-0e55-4cc7-8599-1b6d5d94de02"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:22:10.707841 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.707664 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4f5cb00-0e55-4cc7-8599-1b6d5d94de02-maas-api-tls" pod "e4f5cb00-0e55-4cc7-8599-1b6d5d94de02" (UID: "e4f5cb00-0e55-4cc7-8599-1b6d5d94de02"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:22:10.708059 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.708027 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4f5cb00-0e55-4cc7-8599-1b6d5d94de02-kube-api-access-8pmct" pod "e4f5cb00-0e55-4cc7-8599-1b6d5d94de02" (UID: "e4f5cb00-0e55-4cc7-8599-1b6d5d94de02"). InnerVolumeSpecName "kube-api-access-8pmct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:22:10.708059 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.708041 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4f5cb00-0e55-4cc7-8599-1b6d5d94de02-metrics-certs" pod "e4f5cb00-0e55-4cc7-8599-1b6d5d94de02" (UID: "e4f5cb00-0e55-4cc7-8599-1b6d5d94de02"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:22:10.869133 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.806080 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450-plugins-config-volume\") pod \"d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450\" (UID: \"d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450\") " Oct 30 07:22:10.869133 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.806375 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450-kube-api-access-zfrsv\" (UniqueName: \"kubernetes.io/projected/d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450-kube-api-access-zfrsv\") pod \"d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450\" (UID: \"d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450\") " Oct 30 07:22:10.869133 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.806986 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4f5cb00-0e55-4cc7-8599-1b6d5d94de02-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:10.869133 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.807010 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4f5cb00-0e55-4cc7-8599-1b6d5d94de02-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:10.869133 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.807025 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pmct\" (UniqueName: \"kubernetes.io/projected/e4f5cb00-0e55-4cc7-8599-1b6d5d94de02-kube-api-access-8pmct\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:10.869133 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.807038 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4f5cb00-0e55-4cc7-8599-1b6d5d94de02-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:10.869133 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.833951 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450-plugins-config-volume" pod "d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450" (UID: "d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:22:10.931505 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:10.876046 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:22:11.000948 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:11.000597 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:22:11.003267 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:11.003176 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450-kube-api-access-zfrsv" pod "d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450" (UID: "d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450"). InnerVolumeSpecName "kube-api-access-zfrsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:22:11.094921 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:11.066863 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-794fc67c4c-z2hjz" Oct 30 07:22:11.094921 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:11.090071 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfrsv\" (UniqueName: \"kubernetes.io/projected/d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450-kube-api-access-zfrsv\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:11.094921 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:11.090282 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:11.163423 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:22:11.163327 4951 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f5cb00_0e55_4cc7_8599_1b6d5d94de02.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f5cb00_0e55_4cc7_8599_1b6d5d94de02.slice/crio-c7fa307ac884221ea3feb0029c28b711c2fba2a1f74c4c1bf8ae7f6a5f06571c\": RecentStats: unable to find data in memory cache]" Oct 30 07:22:11.191005 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:11.190969 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a995882-9616-44c9-8f3c-2d6615d7bbde-kube-api-access-r7npg\" (UniqueName: \"kubernetes.io/projected/4a995882-9616-44c9-8f3c-2d6615d7bbde-kube-api-access-r7npg\") pod \"4a995882-9616-44c9-8f3c-2d6615d7bbde\" (UID: \"4a995882-9616-44c9-8f3c-2d6615d7bbde\") " Oct 30 07:22:11.191195 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:11.191053 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a995882-9616-44c9-8f3c-2d6615d7bbde-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a995882-9616-44c9-8f3c-2d6615d7bbde-plugins-config-volume\") pod \"4a995882-9616-44c9-8f3c-2d6615d7bbde\" (UID: \"4a995882-9616-44c9-8f3c-2d6615d7bbde\") " Oct 30 07:22:11.191533 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:11.191504 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a995882-9616-44c9-8f3c-2d6615d7bbde-plugins-config-volume" pod "4a995882-9616-44c9-8f3c-2d6615d7bbde" (UID: "4a995882-9616-44c9-8f3c-2d6615d7bbde"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:22:11.191666 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:11.191639 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a995882-9616-44c9-8f3c-2d6615d7bbde-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:11.193400 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:11.193367 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a995882-9616-44c9-8f3c-2d6615d7bbde-kube-api-access-r7npg" pod "4a995882-9616-44c9-8f3c-2d6615d7bbde" (UID: "4a995882-9616-44c9-8f3c-2d6615d7bbde"). InnerVolumeSpecName "kube-api-access-r7npg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:22:11.280650 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:11.277764 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4"] Oct 30 07:22:11.289085 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:11.285344 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0cd8f0-7c6cfb6f4c-ldjd4"] Oct 30 07:22:11.293253 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:11.293217 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7npg\" (UniqueName: \"kubernetes.io/projected/4a995882-9616-44c9-8f3c-2d6615d7bbde-kube-api-access-r7npg\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:11.644208 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:11.644183 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-6bbb96bf6d-wwc5q" Oct 30 07:22:11.644208 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:11.644199 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-6bbb96bf6d-wwc5q" event={"ID":"d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450","Type":"ContainerDied","Data":"06c9243981fff5efbe24f51001c0ff8479f3d468a84fa3f5f1054f2125c137b7"} Oct 30 07:22:11.644472 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:11.644265 4951 scope.go:122] "RemoveContainer" containerID="1fb10342388d7c014b375457cc3f79d06ddf3b48b57ea64a8af53f0211b6bb76" Oct 30 07:22:11.645643 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:11.645534 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-794fc67c4c-z2hjz" event={"ID":"4a995882-9616-44c9-8f3c-2d6615d7bbde","Type":"ContainerDied","Data":"53bff5b202c5943bbf492f02a96adcd88a7eafb6c47e001d4af33da41756dc8e"} Oct 30 07:22:11.645643 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:11.645628 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-794fc67c4c-z2hjz" Oct 30 07:22:11.656088 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:11.656071 4951 scope.go:122] "RemoveContainer" containerID="cd75c1b7ea8ceef193e78de57f949a9d189c471cb866fa82ccafc4fdf3c86d70" Oct 30 07:22:11.698664 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:11.698630 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-794fc67c4c-z2hjz"] Oct 30 07:22:11.702879 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:11.702852 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0cd8f0-794fc67c4c-z2hjz"] Oct 30 07:22:11.736659 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:11.736632 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-6bbb96bf6d-wwc5q"] Oct 30 07:22:11.740434 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:11.740408 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0cd8f0-6bbb96bf6d-wwc5q"] Oct 30 07:22:11.772166 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:11.772143 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a995882-9616-44c9-8f3c-2d6615d7bbde" path="/var/lib/kubelet/pods/4a995882-9616-44c9-8f3c-2d6615d7bbde/volumes" Oct 30 07:22:11.772625 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:11.772609 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450" path="/var/lib/kubelet/pods/d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450/volumes" Oct 30 07:22:11.773050 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:11.773036 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4f5cb00-0e55-4cc7-8599-1b6d5d94de02" path="/var/lib/kubelet/pods/e4f5cb00-0e55-4cc7-8599-1b6d5d94de02/volumes" Oct 30 07:22:13.520780 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:13.520748 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-58bf9bd7f9-7m5cm" Oct 30 07:22:13.521417 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:13.520825 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-8c45f755d-2tr2x" Oct 30 07:22:17.998152 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:17.998120 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5"] Oct 30 07:22:17.998651 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:17.998387 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" podUID="2898e6e0-5229-4c67-b3e3-8f2076284b86" containerName="istio-proxy" containerID="cri-o://b570b4b4afd8fa284d9ebf9664956f3376d292c4eae79b456106c29d36dd0348" gracePeriod=30 Oct 30 07:22:18.409345 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.409314 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h"] Oct 30 07:22:18.409981 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.409963 4951 cpu_manager.go:414] "containerMap: removing container" podUID="4a995882-9616-44c9-8f3c-2d6615d7bbde" containerName="payload-pre-processing" Oct 30 07:22:18.409981 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.409981 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a995882-9616-44c9-8f3c-2d6615d7bbde" containerName="payload-pre-processing" Oct 30 07:22:18.410135 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.410022 4951 cpu_manager.go:414] "containerMap: removing container" podUID="e4f5cb00-0e55-4cc7-8599-1b6d5d94de02" containerName="maas-api" Oct 30 07:22:18.410135 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.410028 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f5cb00-0e55-4cc7-8599-1b6d5d94de02" containerName="maas-api" Oct 30 07:22:18.410135 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.410039 4951 cpu_manager.go:414] "containerMap: removing container" podUID="d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450" containerName="payload-processing" Oct 30 07:22:18.410135 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.410044 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450" containerName="payload-processing" Oct 30 07:22:18.410348 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.410141 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4f5cb00-0e55-4cc7-8599-1b6d5d94de02" containerName="maas-api" Oct 30 07:22:18.410348 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.410149 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2fda5b5-fcf9-4377-8b53-4e9ffa0a6450" containerName="payload-processing" Oct 30 07:22:18.410348 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.410161 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a995882-9616-44c9-8f3c-2d6615d7bbde" containerName="payload-pre-processing" Oct 30 07:22:18.413634 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.413609 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:22:18.432127 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.432085 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-597d57-openshift-default-dockercfg-jxxkz\"" Oct 30 07:22:18.449843 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.449814 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h"] Oct 30 07:22:18.528364 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.528319 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f345a13-140a-4122-9ad1-ece45c401b58-istio-token\") pod \"e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:22:18.528536 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.528442 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-workload-certs\") pod \"e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:22:18.528536 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.528475 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-istio-envoy\") pod \"e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:22:18.528675 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.528542 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-istio-data\") pod \"e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:22:18.528675 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.528590 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-credential-socket\") pod \"e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:22:18.528803 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.528782 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f345a13-140a-4122-9ad1-ece45c401b58-istiod-ca-cert\") pod \"e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:22:18.528884 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.528872 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f345a13-140a-4122-9ad1-ece45c401b58-istio-podinfo\") pod \"e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:22:18.528972 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.528898 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-workload-socket\") pod \"e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:22:18.529031 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.528996 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp8xz\" (UniqueName: \"kubernetes.io/projected/4f345a13-140a-4122-9ad1-ece45c401b58-kube-api-access-kp8xz\") pod \"e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:22:18.632625 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.632553 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f345a13-140a-4122-9ad1-ece45c401b58-istiod-ca-cert\") pod \"e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:22:18.632946 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.632926 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f345a13-140a-4122-9ad1-ece45c401b58-istio-podinfo\") pod \"e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:22:18.633088 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.633072 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-workload-socket\") pod \"e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:22:18.633304 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.633289 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp8xz\" (UniqueName: \"kubernetes.io/projected/4f345a13-140a-4122-9ad1-ece45c401b58-kube-api-access-kp8xz\") pod \"e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:22:18.633415 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.633399 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f345a13-140a-4122-9ad1-ece45c401b58-istio-token\") pod \"e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:22:18.633553 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.633539 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-workload-certs\") pod \"e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:22:18.633687 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.633672 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-istio-envoy\") pod \"e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:22:18.639604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.633802 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-istio-data\") pod \"e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:22:18.658108 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.658070 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-credential-socket\") pod \"e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:22:18.658622 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.658595 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f345a13-140a-4122-9ad1-ece45c401b58-istiod-ca-cert\") pod \"e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:22:18.658772 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.650915 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-workload-certs\") pod \"e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:22:18.801374 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.801339 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-istio-envoy\") pod \"e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:22:18.802871 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.802835 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-workload-socket\") pod \"e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:22:18.803070 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.803052 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-credential-socket\") pod \"e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:22:18.803259 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.803235 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-istio-data\") pod \"e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:22:18.833506 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.832302 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp8xz\" (UniqueName: \"kubernetes.io/projected/4f345a13-140a-4122-9ad1-ece45c401b58-kube-api-access-kp8xz\") pod \"e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:22:18.833506 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.833350 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f345a13-140a-4122-9ad1-ece45c401b58-istio-podinfo\") pod \"e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:22:18.875041 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.873716 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f345a13-140a-4122-9ad1-ece45c401b58-istio-token\") pod \"e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:22:18.882931 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.882351 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bdd496-55f7c4fcdd-v4qkn"] Oct 30 07:22:18.891799 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:18.891502 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-7cd9fbdcff-zn75b"] Oct 30 07:22:19.043257 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.043209 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:22:19.074032 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.074002 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2"] Oct 30 07:22:19.074469 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.074437 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2" podUID="258478b4-7854-48a9-8138-707475c24418" containerName="maas-api" containerID="cri-o://5f3967617cc02bc0e3b794513de12da7ec894e06e3564beb7957b6336f6f10b2" gracePeriod=30 Oct 30 07:22:19.274635 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.272754 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-438855-5cf54f6f44-wbkqt"] Oct 30 07:22:19.274635 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.273879 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-5cf54f6f44-wbkqt" podUID="923bebff-1573-4dbc-997c-24a416579f99" containerName="payload-processing" containerID="cri-o://4837b93a949ed6b52f3ec1891b238040b1a31bc75367f0ca817fffe9f949625e" gracePeriod=30 Oct 30 07:22:19.388060 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.388037 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2" Oct 30 07:22:19.413232 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.413196 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h"] Oct 30 07:22:19.416401 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:22:19.416360 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f345a13_140a_4122_9ad1_ece45c401b58.slice/crio-c536caabb5f9910cb1ab27d6c912e88a0247129059617f679f4e9ba9cb0b0332 WatchSource:0}: Error finding container c536caabb5f9910cb1ab27d6c912e88a0247129059617f679f4e9ba9cb0b0332: Status 404 returned error can't find the container with id c536caabb5f9910cb1ab27d6c912e88a0247129059617f679f4e9ba9cb0b0332 Oct 30 07:22:19.419086 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.419041 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:22:19.419211 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.419133 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:22:19.419211 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.419178 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:22:19.502994 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.502554 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/258478b4-7854-48a9-8138-707475c24418-metrics-certs\" (UniqueName: \"kubernetes.io/secret/258478b4-7854-48a9-8138-707475c24418-metrics-certs\") pod \"258478b4-7854-48a9-8138-707475c24418\" (UID: \"258478b4-7854-48a9-8138-707475c24418\") " Oct 30 07:22:19.503179 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.503119 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/258478b4-7854-48a9-8138-707475c24418-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/258478b4-7854-48a9-8138-707475c24418-maas-api-tls\") pod \"258478b4-7854-48a9-8138-707475c24418\" (UID: \"258478b4-7854-48a9-8138-707475c24418\") " Oct 30 07:22:19.503650 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.503610 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/258478b4-7854-48a9-8138-707475c24418-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/258478b4-7854-48a9-8138-707475c24418-openshift-service-ca\") pod \"258478b4-7854-48a9-8138-707475c24418\" (UID: \"258478b4-7854-48a9-8138-707475c24418\") " Oct 30 07:22:19.503797 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.503781 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/258478b4-7854-48a9-8138-707475c24418-kube-api-access-dgl4b\" (UniqueName: \"kubernetes.io/projected/258478b4-7854-48a9-8138-707475c24418-kube-api-access-dgl4b\") pod \"258478b4-7854-48a9-8138-707475c24418\" (UID: \"258478b4-7854-48a9-8138-707475c24418\") " Oct 30 07:22:19.504735 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.504165 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/258478b4-7854-48a9-8138-707475c24418-openshift-service-ca" pod "258478b4-7854-48a9-8138-707475c24418" (UID: "258478b4-7854-48a9-8138-707475c24418"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:22:19.504735 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.504687 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/258478b4-7854-48a9-8138-707475c24418-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:19.506503 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.506477 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/258478b4-7854-48a9-8138-707475c24418-maas-api-tls" pod "258478b4-7854-48a9-8138-707475c24418" (UID: "258478b4-7854-48a9-8138-707475c24418"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:22:19.507716 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.507680 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/258478b4-7854-48a9-8138-707475c24418-metrics-certs" pod "258478b4-7854-48a9-8138-707475c24418" (UID: "258478b4-7854-48a9-8138-707475c24418"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:22:19.509033 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.508991 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/258478b4-7854-48a9-8138-707475c24418-kube-api-access-dgl4b" pod "258478b4-7854-48a9-8138-707475c24418" (UID: "258478b4-7854-48a9-8138-707475c24418"). InnerVolumeSpecName "kube-api-access-dgl4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:22:19.565141 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.565108 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-5cf54f6f44-wbkqt" Oct 30 07:22:19.612903 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.611116 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgl4b\" (UniqueName: \"kubernetes.io/projected/258478b4-7854-48a9-8138-707475c24418-kube-api-access-dgl4b\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:19.631645 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.630558 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/258478b4-7854-48a9-8138-707475c24418-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:19.631645 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.630634 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/258478b4-7854-48a9-8138-707475c24418-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:19.737635 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.737599 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/923bebff-1573-4dbc-997c-24a416579f99-plugins-config-volume" pod "923bebff-1573-4dbc-997c-24a416579f99" (UID: "923bebff-1573-4dbc-997c-24a416579f99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:22:19.739841 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.739783 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/923bebff-1573-4dbc-997c-24a416579f99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/923bebff-1573-4dbc-997c-24a416579f99-plugins-config-volume\") pod \"923bebff-1573-4dbc-997c-24a416579f99\" (UID: \"923bebff-1573-4dbc-997c-24a416579f99\") " Oct 30 07:22:19.755113 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.755079 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/923bebff-1573-4dbc-997c-24a416579f99-kube-api-access-tgg6g\" (UniqueName: \"kubernetes.io/projected/923bebff-1573-4dbc-997c-24a416579f99-kube-api-access-tgg6g\") pod \"923bebff-1573-4dbc-997c-24a416579f99\" (UID: \"923bebff-1573-4dbc-997c-24a416579f99\") " Oct 30 07:22:19.755980 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.755954 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/923bebff-1573-4dbc-997c-24a416579f99-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:19.813901 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.813749 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/923bebff-1573-4dbc-997c-24a416579f99-kube-api-access-tgg6g" pod "923bebff-1573-4dbc-997c-24a416579f99" (UID: "923bebff-1573-4dbc-997c-24a416579f99"). InnerVolumeSpecName "kube-api-access-tgg6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:22:19.872622 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.872589 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgg6g\" (UniqueName: \"kubernetes.io/projected/923bebff-1573-4dbc-997c-24a416579f99-kube-api-access-tgg6g\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:19.909393 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.909357 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" event={"ID":"4f345a13-140a-4122-9ad1-ece45c401b58","Type":"ContainerStarted","Data":"285427ad353b770ac9cbaa25b34fcaae40926d5fa3e5bc9f62d634e61054a8bd"} Oct 30 07:22:19.909698 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.909672 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" event={"ID":"4f345a13-140a-4122-9ad1-ece45c401b58","Type":"ContainerStarted","Data":"c536caabb5f9910cb1ab27d6c912e88a0247129059617f679f4e9ba9cb0b0332"} Oct 30 07:22:19.911073 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.911053 4951 generic.go:356] "Generic (PLEG): container finished" podID="258478b4-7854-48a9-8138-707475c24418" containerID="5f3967617cc02bc0e3b794513de12da7ec894e06e3564beb7957b6336f6f10b2" exitCode=0 Oct 30 07:22:19.911181 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.911115 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2" Oct 30 07:22:19.911181 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.911127 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2" event={"ID":"258478b4-7854-48a9-8138-707475c24418","Type":"ContainerDied","Data":"5f3967617cc02bc0e3b794513de12da7ec894e06e3564beb7957b6336f6f10b2"} Oct 30 07:22:19.911181 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.911179 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2" event={"ID":"258478b4-7854-48a9-8138-707475c24418","Type":"ContainerDied","Data":"89a57af86a95eff36b7f1ae7c1cbba259798a0f45b1f0a84a3d2d8e29ae1cab4"} Oct 30 07:22:19.911381 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.911209 4951 scope.go:122] "RemoveContainer" containerID="5f3967617cc02bc0e3b794513de12da7ec894e06e3564beb7957b6336f6f10b2" Oct 30 07:22:19.912820 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.912788 4951 generic.go:356] "Generic (PLEG): container finished" podID="923bebff-1573-4dbc-997c-24a416579f99" containerID="4837b93a949ed6b52f3ec1891b238040b1a31bc75367f0ca817fffe9f949625e" exitCode=0 Oct 30 07:22:19.913003 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.912976 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-5cf54f6f44-wbkqt" event={"ID":"923bebff-1573-4dbc-997c-24a416579f99","Type":"ContainerDied","Data":"4837b93a949ed6b52f3ec1891b238040b1a31bc75367f0ca817fffe9f949625e"} Oct 30 07:22:19.913116 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.913036 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-5cf54f6f44-wbkqt" event={"ID":"923bebff-1573-4dbc-997c-24a416579f99","Type":"ContainerDied","Data":"2ed9ccd07f09557bbc5ea988cb65d41fa9fc2bc77766e04205d41c89f5eb53ff"} Oct 30 07:22:19.913116 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.913112 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-438855-5cf54f6f44-wbkqt" Oct 30 07:22:19.925457 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.925438 4951 scope.go:122] "RemoveContainer" containerID="5f3967617cc02bc0e3b794513de12da7ec894e06e3564beb7957b6336f6f10b2" Oct 30 07:22:19.925753 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:22:19.925731 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f3967617cc02bc0e3b794513de12da7ec894e06e3564beb7957b6336f6f10b2\": container with ID starting with 5f3967617cc02bc0e3b794513de12da7ec894e06e3564beb7957b6336f6f10b2 not found: ID does not exist" containerID="5f3967617cc02bc0e3b794513de12da7ec894e06e3564beb7957b6336f6f10b2" Oct 30 07:22:19.925861 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.925758 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f3967617cc02bc0e3b794513de12da7ec894e06e3564beb7957b6336f6f10b2"} err="failed to get container status \"5f3967617cc02bc0e3b794513de12da7ec894e06e3564beb7957b6336f6f10b2\": rpc error: code = NotFound desc = could not find container \"5f3967617cc02bc0e3b794513de12da7ec894e06e3564beb7957b6336f6f10b2\": container with ID starting with 5f3967617cc02bc0e3b794513de12da7ec894e06e3564beb7957b6336f6f10b2 not found: ID does not exist" Oct 30 07:22:19.925861 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.925778 4951 scope.go:122] "RemoveContainer" containerID="4837b93a949ed6b52f3ec1891b238040b1a31bc75367f0ca817fffe9f949625e" Oct 30 07:22:19.937968 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.937949 4951 scope.go:122] "RemoveContainer" containerID="4837b93a949ed6b52f3ec1891b238040b1a31bc75367f0ca817fffe9f949625e" Oct 30 07:22:19.938280 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:22:19.938260 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4837b93a949ed6b52f3ec1891b238040b1a31bc75367f0ca817fffe9f949625e\": container with ID starting with 4837b93a949ed6b52f3ec1891b238040b1a31bc75367f0ca817fffe9f949625e not found: ID does not exist" containerID="4837b93a949ed6b52f3ec1891b238040b1a31bc75367f0ca817fffe9f949625e" Oct 30 07:22:19.938383 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.938283 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4837b93a949ed6b52f3ec1891b238040b1a31bc75367f0ca817fffe9f949625e"} err="failed to get container status \"4837b93a949ed6b52f3ec1891b238040b1a31bc75367f0ca817fffe9f949625e\": rpc error: code = NotFound desc = could not find container \"4837b93a949ed6b52f3ec1891b238040b1a31bc75367f0ca817fffe9f949625e\": container with ID starting with 4837b93a949ed6b52f3ec1891b238040b1a31bc75367f0ca817fffe9f949625e not found: ID does not exist" Oct 30 07:22:19.958048 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:22:19.958016 4951 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod923bebff_1573_4dbc_997c_24a416579f99.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod258478b4_7854_48a9_8138_707475c24418.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod258478b4_7854_48a9_8138_707475c24418.slice/crio-89a57af86a95eff36b7f1ae7c1cbba259798a0f45b1f0a84a3d2d8e29ae1cab4\": RecentStats: unable to find data in memory cache]" Oct 30 07:22:19.959798 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:19.959759 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" podStartSLOduration=1.9597457999999999 podStartE2EDuration="1.9597458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:22:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:22:19.95565875 +0000 UTC m=+1546.298706750" watchObservedRunningTime="2026-10-30 07:22:19.9597458 +0000 UTC m=+1546.302793765" Oct 30 07:22:20.000633 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:20.000596 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-438855-5cf54f6f44-wbkqt"] Oct 30 07:22:20.012025 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:20.011989 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-438855-5cf54f6f44-wbkqt"] Oct 30 07:22:20.043874 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:20.043826 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:22:20.045315 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:20.045289 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2"] Oct 30 07:22:20.046326 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:20.046297 4951 patch_prober.go:29] interesting pod/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 07:22:20.046479 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:20.046367 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" podUID="4f345a13-140a-4122-9ad1-ece45c401b58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 07:22:20.048879 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:20.048856 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-438855-fb469bcf5-gsxk2"] Oct 30 07:22:21.048745 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:21.048715 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:22:21.773490 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:21.773458 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="258478b4-7854-48a9-8138-707475c24418" path="/var/lib/kubelet/pods/258478b4-7854-48a9-8138-707475c24418/volumes" Oct 30 07:22:21.774009 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:21.773989 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="923bebff-1573-4dbc-997c-24a416579f99" path="/var/lib/kubelet/pods/923bebff-1573-4dbc-997c-24a416579f99/volumes" Oct 30 07:22:21.931123 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:21.931088 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:22:21.932160 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:21.932138 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:22:23.031503 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:22:23.031430 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:22:23.277106 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:23.277046 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:22:23.330304 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:23.330270 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-workload-socket\") pod \"2898e6e0-5229-4c67-b3e3-8f2076284b86\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " Oct 30 07:22:23.330472 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:23.330361 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-istio-data\") pod \"2898e6e0-5229-4c67-b3e3-8f2076284b86\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " Oct 30 07:22:23.330472 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:23.330424 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2898e6e0-5229-4c67-b3e3-8f2076284b86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2898e6e0-5229-4c67-b3e3-8f2076284b86-istio-podinfo\") pod \"2898e6e0-5229-4c67-b3e3-8f2076284b86\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " Oct 30 07:22:23.330472 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:23.330455 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2898e6e0-5229-4c67-b3e3-8f2076284b86-istio-token\" (UniqueName: \"kubernetes.io/projected/2898e6e0-5229-4c67-b3e3-8f2076284b86-istio-token\") pod \"2898e6e0-5229-4c67-b3e3-8f2076284b86\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " Oct 30 07:22:23.330639 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:23.330562 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-workload-socket" pod "2898e6e0-5229-4c67-b3e3-8f2076284b86" (UID: "2898e6e0-5229-4c67-b3e3-8f2076284b86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:22:23.330693 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:23.330646 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-istio-data" pod "2898e6e0-5229-4c67-b3e3-8f2076284b86" (UID: "2898e6e0-5229-4c67-b3e3-8f2076284b86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:22:23.330750 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:23.330705 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2898e6e0-5229-4c67-b3e3-8f2076284b86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2898e6e0-5229-4c67-b3e3-8f2076284b86-istiod-ca-cert\") pod \"2898e6e0-5229-4c67-b3e3-8f2076284b86\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " Oct 30 07:22:23.330827 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:23.330811 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-credential-socket\") pod \"2898e6e0-5229-4c67-b3e3-8f2076284b86\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " Oct 30 07:22:23.330886 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:23.330854 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2898e6e0-5229-4c67-b3e3-8f2076284b86-kube-api-access-5h2x7\" (UniqueName: \"kubernetes.io/projected/2898e6e0-5229-4c67-b3e3-8f2076284b86-kube-api-access-5h2x7\") pod \"2898e6e0-5229-4c67-b3e3-8f2076284b86\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " Oct 30 07:22:23.330941 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:23.330902 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-istio-envoy\") pod \"2898e6e0-5229-4c67-b3e3-8f2076284b86\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " Oct 30 07:22:23.330991 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:23.330940 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-workload-certs\") pod \"2898e6e0-5229-4c67-b3e3-8f2076284b86\" (UID: \"2898e6e0-5229-4c67-b3e3-8f2076284b86\") " Oct 30 07:22:23.331065 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:23.331044 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-credential-socket" pod "2898e6e0-5229-4c67-b3e3-8f2076284b86" (UID: "2898e6e0-5229-4c67-b3e3-8f2076284b86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:22:23.331352 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:23.331329 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-workload-certs" pod "2898e6e0-5229-4c67-b3e3-8f2076284b86" (UID: "2898e6e0-5229-4c67-b3e3-8f2076284b86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:22:23.331450 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:23.331359 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2898e6e0-5229-4c67-b3e3-8f2076284b86-istiod-ca-cert" pod "2898e6e0-5229-4c67-b3e3-8f2076284b86" (UID: "2898e6e0-5229-4c67-b3e3-8f2076284b86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:22:23.331641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:23.331625 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:23.331641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:23.331642 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:23.331794 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:23.331652 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2898e6e0-5229-4c67-b3e3-8f2076284b86-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:23.331794 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:23.331661 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:23.331794 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:23.331671 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:23.333183 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:23.333159 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2898e6e0-5229-4c67-b3e3-8f2076284b86-istio-podinfo" pod "2898e6e0-5229-4c67-b3e3-8f2076284b86" (UID: "2898e6e0-5229-4c67-b3e3-8f2076284b86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:22:23.333183 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:23.333175 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2898e6e0-5229-4c67-b3e3-8f2076284b86-kube-api-access-5h2x7" pod "2898e6e0-5229-4c67-b3e3-8f2076284b86" (UID: "2898e6e0-5229-4c67-b3e3-8f2076284b86"). InnerVolumeSpecName "kube-api-access-5h2x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:22:23.333406 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:23.333388 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2898e6e0-5229-4c67-b3e3-8f2076284b86-istio-token" pod "2898e6e0-5229-4c67-b3e3-8f2076284b86" (UID: "2898e6e0-5229-4c67-b3e3-8f2076284b86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:22:23.334965 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:23.334929 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-istio-envoy" pod "2898e6e0-5229-4c67-b3e3-8f2076284b86" (UID: "2898e6e0-5229-4c67-b3e3-8f2076284b86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:22:23.432947 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:23.432911 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2898e6e0-5229-4c67-b3e3-8f2076284b86-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:23.432947 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:23.432942 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2898e6e0-5229-4c67-b3e3-8f2076284b86-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:23.432947 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:23.432952 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h2x7\" (UniqueName: \"kubernetes.io/projected/2898e6e0-5229-4c67-b3e3-8f2076284b86-kube-api-access-5h2x7\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:23.433196 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:23.432963 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2898e6e0-5229-4c67-b3e3-8f2076284b86-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:23.886593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:23.884813 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-55f7c4fcdd-v4qkn" podUID="20ccf7cf-223b-48aa-8845-af23df3d1c29" containerName="payload-processing" containerID="cri-o://aa46f4d423c666abdd53e8f378dd057d6da1bb47ab561f24ada9032f4fd7568d" gracePeriod=25 Oct 30 07:22:23.901064 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:23.894513 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-7cd9fbdcff-zn75b" podUID="d60f718d-86b8-4d75-a28a-7d7ad358cdb5" containerName="payload-pre-processing" containerID="cri-o://3027d20743bdd8d9202d7748b4d497b763418b5e5dff48c9f5f02eb565c7e204" gracePeriod=25 Oct 30 07:22:23.996007 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:23.991927 4951 generic.go:356] "Generic (PLEG): container finished" podID="2898e6e0-5229-4c67-b3e3-8f2076284b86" containerID="b570b4b4afd8fa284d9ebf9664956f3376d292c4eae79b456106c29d36dd0348" exitCode=0 Oct 30 07:22:23.996007 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:23.993225 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" Oct 30 07:22:24.010038 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.009996 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" event={"ID":"2898e6e0-5229-4c67-b3e3-8f2076284b86","Type":"ContainerDied","Data":"b570b4b4afd8fa284d9ebf9664956f3376d292c4eae79b456106c29d36dd0348"} Oct 30 07:22:24.010237 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.010091 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5" event={"ID":"2898e6e0-5229-4c67-b3e3-8f2076284b86","Type":"ContainerDied","Data":"be87df6dded6659fcb65ff40d21a5002f65e9b3bcbe5bb8a599fb6bdacb12652"} Oct 30 07:22:24.010237 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.010142 4951 scope.go:122] "RemoveContainer" containerID="b570b4b4afd8fa284d9ebf9664956f3376d292c4eae79b456106c29d36dd0348" Oct 30 07:22:24.120013 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.119985 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5"] Oct 30 07:22:24.136285 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.136165 4951 scope.go:122] "RemoveContainer" containerID="1e43449f9e6c7034fa598e65aef6052612b73a058c0388f7e0fd1239aa6afb21" Oct 30 07:22:24.141371 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.141300 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-0cd8f0-openshift-default-9f9c888cc-h95f5"] Oct 30 07:22:24.159651 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.155350 4951 scope.go:122] "RemoveContainer" containerID="b570b4b4afd8fa284d9ebf9664956f3376d292c4eae79b456106c29d36dd0348" Oct 30 07:22:24.160256 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:22:24.160018 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b570b4b4afd8fa284d9ebf9664956f3376d292c4eae79b456106c29d36dd0348\": container with ID starting with b570b4b4afd8fa284d9ebf9664956f3376d292c4eae79b456106c29d36dd0348 not found: ID does not exist" containerID="b570b4b4afd8fa284d9ebf9664956f3376d292c4eae79b456106c29d36dd0348" Oct 30 07:22:24.160256 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.160064 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b570b4b4afd8fa284d9ebf9664956f3376d292c4eae79b456106c29d36dd0348"} err="failed to get container status \"b570b4b4afd8fa284d9ebf9664956f3376d292c4eae79b456106c29d36dd0348\": rpc error: code = NotFound desc = could not find container \"b570b4b4afd8fa284d9ebf9664956f3376d292c4eae79b456106c29d36dd0348\": container with ID starting with b570b4b4afd8fa284d9ebf9664956f3376d292c4eae79b456106c29d36dd0348 not found: ID does not exist" Oct 30 07:22:24.160256 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.160096 4951 scope.go:122] "RemoveContainer" containerID="1e43449f9e6c7034fa598e65aef6052612b73a058c0388f7e0fd1239aa6afb21" Oct 30 07:22:24.160709 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:22:24.160664 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e43449f9e6c7034fa598e65aef6052612b73a058c0388f7e0fd1239aa6afb21\": container with ID starting with 1e43449f9e6c7034fa598e65aef6052612b73a058c0388f7e0fd1239aa6afb21 not found: ID does not exist" containerID="1e43449f9e6c7034fa598e65aef6052612b73a058c0388f7e0fd1239aa6afb21" Oct 30 07:22:24.160812 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.160702 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e43449f9e6c7034fa598e65aef6052612b73a058c0388f7e0fd1239aa6afb21"} err="failed to get container status \"1e43449f9e6c7034fa598e65aef6052612b73a058c0388f7e0fd1239aa6afb21\": rpc error: code = NotFound desc = could not find container \"1e43449f9e6c7034fa598e65aef6052612b73a058c0388f7e0fd1239aa6afb21\": container with ID starting with 1e43449f9e6c7034fa598e65aef6052612b73a058c0388f7e0fd1239aa6afb21 not found: ID does not exist" Oct 30 07:22:24.382136 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.382107 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-55f7c4fcdd-v4qkn" Oct 30 07:22:24.431678 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.431650 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-7cd9fbdcff-zn75b" Oct 30 07:22:24.499269 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.499231 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d60f718d-86b8-4d75-a28a-7d7ad358cdb5-kube-api-access-mzv62\" (UniqueName: \"kubernetes.io/projected/d60f718d-86b8-4d75-a28a-7d7ad358cdb5-kube-api-access-mzv62\") pod \"d60f718d-86b8-4d75-a28a-7d7ad358cdb5\" (UID: \"d60f718d-86b8-4d75-a28a-7d7ad358cdb5\") " Oct 30 07:22:24.499501 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.499479 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20ccf7cf-223b-48aa-8845-af23df3d1c29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20ccf7cf-223b-48aa-8845-af23df3d1c29-plugins-config-volume\") pod \"20ccf7cf-223b-48aa-8845-af23df3d1c29\" (UID: \"20ccf7cf-223b-48aa-8845-af23df3d1c29\") " Oct 30 07:22:24.499606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.499534 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20ccf7cf-223b-48aa-8845-af23df3d1c29-kube-api-access-2hn6v\" (UniqueName: \"kubernetes.io/projected/20ccf7cf-223b-48aa-8845-af23df3d1c29-kube-api-access-2hn6v\") pod \"20ccf7cf-223b-48aa-8845-af23df3d1c29\" (UID: \"20ccf7cf-223b-48aa-8845-af23df3d1c29\") " Oct 30 07:22:24.499684 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.499566 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d60f718d-86b8-4d75-a28a-7d7ad358cdb5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d60f718d-86b8-4d75-a28a-7d7ad358cdb5-plugins-config-volume\") pod \"d60f718d-86b8-4d75-a28a-7d7ad358cdb5\" (UID: \"d60f718d-86b8-4d75-a28a-7d7ad358cdb5\") " Oct 30 07:22:24.500628 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.500598 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d60f718d-86b8-4d75-a28a-7d7ad358cdb5-plugins-config-volume" pod "d60f718d-86b8-4d75-a28a-7d7ad358cdb5" (UID: "d60f718d-86b8-4d75-a28a-7d7ad358cdb5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:22:24.501408 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.501372 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20ccf7cf-223b-48aa-8845-af23df3d1c29-plugins-config-volume" pod "20ccf7cf-223b-48aa-8845-af23df3d1c29" (UID: "20ccf7cf-223b-48aa-8845-af23df3d1c29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:22:24.503379 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.503350 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d60f718d-86b8-4d75-a28a-7d7ad358cdb5-kube-api-access-mzv62" pod "d60f718d-86b8-4d75-a28a-7d7ad358cdb5" (UID: "d60f718d-86b8-4d75-a28a-7d7ad358cdb5"). InnerVolumeSpecName "kube-api-access-mzv62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:22:24.506190 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.506161 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ccf7cf-223b-48aa-8845-af23df3d1c29-kube-api-access-2hn6v" pod "20ccf7cf-223b-48aa-8845-af23df3d1c29" (UID: "20ccf7cf-223b-48aa-8845-af23df3d1c29"). InnerVolumeSpecName "kube-api-access-2hn6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:22:24.601451 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.601348 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20ccf7cf-223b-48aa-8845-af23df3d1c29-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:24.601451 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.601387 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hn6v\" (UniqueName: \"kubernetes.io/projected/20ccf7cf-223b-48aa-8845-af23df3d1c29-kube-api-access-2hn6v\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:24.601451 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.601402 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d60f718d-86b8-4d75-a28a-7d7ad358cdb5-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:24.601451 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.601417 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzv62\" (UniqueName: \"kubernetes.io/projected/d60f718d-86b8-4d75-a28a-7d7ad358cdb5-kube-api-access-mzv62\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:24.652728 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.652680 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-7fb879d9f5-p2sqj"] Oct 30 07:22:24.653466 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.653437 4951 cpu_manager.go:414] "containerMap: removing container" podUID="20ccf7cf-223b-48aa-8845-af23df3d1c29" containerName="payload-processing" Oct 30 07:22:24.653466 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.653461 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20ccf7cf-223b-48aa-8845-af23df3d1c29" containerName="payload-processing" Oct 30 07:22:24.653723 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.653487 4951 cpu_manager.go:414] "containerMap: removing container" podUID="258478b4-7854-48a9-8138-707475c24418" containerName="maas-api" Oct 30 07:22:24.653723 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.653495 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="258478b4-7854-48a9-8138-707475c24418" containerName="maas-api" Oct 30 07:22:24.653723 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.653540 4951 cpu_manager.go:414] "containerMap: removing container" podUID="2898e6e0-5229-4c67-b3e3-8f2076284b86" containerName="istio-proxy" Oct 30 07:22:24.653723 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.653548 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2898e6e0-5229-4c67-b3e3-8f2076284b86" containerName="istio-proxy" Oct 30 07:22:24.653723 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.653563 4951 cpu_manager.go:414] "containerMap: removing container" podUID="d60f718d-86b8-4d75-a28a-7d7ad358cdb5" containerName="payload-pre-processing" Oct 30 07:22:24.653723 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.653583 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d60f718d-86b8-4d75-a28a-7d7ad358cdb5" containerName="payload-pre-processing" Oct 30 07:22:24.653723 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.653598 4951 cpu_manager.go:414] "containerMap: removing container" podUID="2898e6e0-5229-4c67-b3e3-8f2076284b86" containerName="istio-proxy" Oct 30 07:22:24.653723 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.653605 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2898e6e0-5229-4c67-b3e3-8f2076284b86" containerName="istio-proxy" Oct 30 07:22:24.653723 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.653627 4951 cpu_manager.go:414] "containerMap: removing container" podUID="923bebff-1573-4dbc-997c-24a416579f99" containerName="payload-processing" Oct 30 07:22:24.653723 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.653632 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="923bebff-1573-4dbc-997c-24a416579f99" containerName="payload-processing" Oct 30 07:22:24.654212 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.653770 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="258478b4-7854-48a9-8138-707475c24418" containerName="maas-api" Oct 30 07:22:24.654212 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.653789 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="2898e6e0-5229-4c67-b3e3-8f2076284b86" containerName="istio-proxy" Oct 30 07:22:24.654212 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.653799 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="d60f718d-86b8-4d75-a28a-7d7ad358cdb5" containerName="payload-pre-processing" Oct 30 07:22:24.654212 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.653807 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="2898e6e0-5229-4c67-b3e3-8f2076284b86" containerName="istio-proxy" Oct 30 07:22:24.654212 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.653819 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="923bebff-1573-4dbc-997c-24a416579f99" containerName="payload-processing" Oct 30 07:22:24.654212 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.653829 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="20ccf7cf-223b-48aa-8845-af23df3d1c29" containerName="payload-processing" Oct 30 07:22:24.657966 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.657946 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-7fb879d9f5-p2sqj" Oct 30 07:22:24.673312 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.673282 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-597d57\"" Oct 30 07:22:24.673474 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.673335 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-597d57-dockercfg-djnh4\"" Oct 30 07:22:24.677381 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.677349 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-7fb879d9f5-p2sqj"] Oct 30 07:22:24.758304 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.747118 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d253264-032c-4c43-92d6-340d3abbdeee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-597d57-7fb879d9f5-p2sqj\" (UID: \"6d253264-032c-4c43-92d6-340d3abbdeee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-7fb879d9f5-p2sqj" Oct 30 07:22:24.758304 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.747265 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjqb8\" (UniqueName: \"kubernetes.io/projected/6d253264-032c-4c43-92d6-340d3abbdeee-kube-api-access-xjqb8\") pod \"payload-pre-processing-e2e-ipp-b-597d57-7fb879d9f5-p2sqj\" (UID: \"6d253264-032c-4c43-92d6-340d3abbdeee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-7fb879d9f5-p2sqj" Oct 30 07:22:24.943098 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.939202 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjqb8\" (UniqueName: \"kubernetes.io/projected/6d253264-032c-4c43-92d6-340d3abbdeee-kube-api-access-xjqb8\") pod \"payload-pre-processing-e2e-ipp-b-597d57-7fb879d9f5-p2sqj\" (UID: \"6d253264-032c-4c43-92d6-340d3abbdeee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-7fb879d9f5-p2sqj" Oct 30 07:22:24.966175 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.966140 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d253264-032c-4c43-92d6-340d3abbdeee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-597d57-7fb879d9f5-p2sqj\" (UID: \"6d253264-032c-4c43-92d6-340d3abbdeee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-7fb879d9f5-p2sqj" Oct 30 07:22:24.967376 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.967338 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d253264-032c-4c43-92d6-340d3abbdeee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-597d57-7fb879d9f5-p2sqj\" (UID: \"6d253264-032c-4c43-92d6-340d3abbdeee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-7fb879d9f5-p2sqj" Oct 30 07:22:24.971040 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.971017 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-597d57-6b875bfb88-trjgk"] Oct 30 07:22:24.974248 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.974223 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjqb8\" (UniqueName: \"kubernetes.io/projected/6d253264-032c-4c43-92d6-340d3abbdeee-kube-api-access-xjqb8\") pod \"payload-pre-processing-e2e-ipp-b-597d57-7fb879d9f5-p2sqj\" (UID: \"6d253264-032c-4c43-92d6-340d3abbdeee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-7fb879d9f5-p2sqj" Oct 30 07:22:24.975504 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.975462 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-7fb879d9f5-p2sqj" Oct 30 07:22:24.977414 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.977376 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-597d57-6b875bfb88-trjgk"] Oct 30 07:22:24.977515 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:24.977458 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-6b875bfb88-trjgk" Oct 30 07:22:25.009532 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:25.008999 4951 generic.go:356] "Generic (PLEG): container finished" podID="d60f718d-86b8-4d75-a28a-7d7ad358cdb5" containerID="3027d20743bdd8d9202d7748b4d497b763418b5e5dff48c9f5f02eb565c7e204" exitCode=0 Oct 30 07:22:25.009532 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:25.009248 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-7cd9fbdcff-zn75b" event={"ID":"d60f718d-86b8-4d75-a28a-7d7ad358cdb5","Type":"ContainerDied","Data":"3027d20743bdd8d9202d7748b4d497b763418b5e5dff48c9f5f02eb565c7e204"} Oct 30 07:22:25.009532 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:25.009308 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-7cd9fbdcff-zn75b" event={"ID":"d60f718d-86b8-4d75-a28a-7d7ad358cdb5","Type":"ContainerDied","Data":"f2845de71c9af4eb7afe61cb5e18eb27c7e41e707324547569fffe98d86f3d38"} Oct 30 07:22:25.009532 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:25.009339 4951 scope.go:122] "RemoveContainer" containerID="3027d20743bdd8d9202d7748b4d497b763418b5e5dff48c9f5f02eb565c7e204" Oct 30 07:22:25.010602 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:25.009945 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-7cd9fbdcff-zn75b" Oct 30 07:22:25.055902 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:25.051472 4951 generic.go:356] "Generic (PLEG): container finished" podID="20ccf7cf-223b-48aa-8845-af23df3d1c29" containerID="aa46f4d423c666abdd53e8f378dd057d6da1bb47ab561f24ada9032f4fd7568d" exitCode=0 Oct 30 07:22:25.055902 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:25.051721 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-55f7c4fcdd-v4qkn" event={"ID":"20ccf7cf-223b-48aa-8845-af23df3d1c29","Type":"ContainerDied","Data":"aa46f4d423c666abdd53e8f378dd057d6da1bb47ab561f24ada9032f4fd7568d"} Oct 30 07:22:25.055902 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:25.051781 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-55f7c4fcdd-v4qkn" event={"ID":"20ccf7cf-223b-48aa-8845-af23df3d1c29","Type":"ContainerDied","Data":"f042d3b360c6d0863507fc6acda43ac6907ca8b2625fe78b3665180d4f1787e5"} Oct 30 07:22:25.055902 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:25.051892 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-55f7c4fcdd-v4qkn" Oct 30 07:22:25.070056 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:25.070017 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvvt2\" (UniqueName: \"kubernetes.io/projected/274fd0e0-2519-4369-8e34-341772f6a37d-kube-api-access-bvvt2\") pod \"payload-processing-e2e-ipp-b-597d57-6b875bfb88-trjgk\" (UID: \"274fd0e0-2519-4369-8e34-341772f6a37d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-6b875bfb88-trjgk" Oct 30 07:22:25.071684 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:25.071655 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/274fd0e0-2519-4369-8e34-341772f6a37d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-597d57-6b875bfb88-trjgk\" (UID: \"274fd0e0-2519-4369-8e34-341772f6a37d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-6b875bfb88-trjgk" Oct 30 07:22:25.131626 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:25.130398 4951 scope.go:122] "RemoveContainer" containerID="3027d20743bdd8d9202d7748b4d497b763418b5e5dff48c9f5f02eb565c7e204" Oct 30 07:22:25.131626 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:22:25.131218 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3027d20743bdd8d9202d7748b4d497b763418b5e5dff48c9f5f02eb565c7e204\": container with ID starting with 3027d20743bdd8d9202d7748b4d497b763418b5e5dff48c9f5f02eb565c7e204 not found: ID does not exist" containerID="3027d20743bdd8d9202d7748b4d497b763418b5e5dff48c9f5f02eb565c7e204" Oct 30 07:22:25.131626 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:25.131257 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3027d20743bdd8d9202d7748b4d497b763418b5e5dff48c9f5f02eb565c7e204"} err="failed to get container status \"3027d20743bdd8d9202d7748b4d497b763418b5e5dff48c9f5f02eb565c7e204\": rpc error: code = NotFound desc = could not find container \"3027d20743bdd8d9202d7748b4d497b763418b5e5dff48c9f5f02eb565c7e204\": container with ID starting with 3027d20743bdd8d9202d7748b4d497b763418b5e5dff48c9f5f02eb565c7e204 not found: ID does not exist" Oct 30 07:22:25.131626 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:25.131282 4951 scope.go:122] "RemoveContainer" containerID="aa46f4d423c666abdd53e8f378dd057d6da1bb47ab561f24ada9032f4fd7568d" Oct 30 07:22:25.162720 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:25.162688 4951 scope.go:122] "RemoveContainer" containerID="aa46f4d423c666abdd53e8f378dd057d6da1bb47ab561f24ada9032f4fd7568d" Oct 30 07:22:25.170433 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:25.170214 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-7cd9fbdcff-zn75b"] Oct 30 07:22:25.171333 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:22:25.170898 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa46f4d423c666abdd53e8f378dd057d6da1bb47ab561f24ada9032f4fd7568d\": container with ID starting with aa46f4d423c666abdd53e8f378dd057d6da1bb47ab561f24ada9032f4fd7568d not found: ID does not exist" containerID="aa46f4d423c666abdd53e8f378dd057d6da1bb47ab561f24ada9032f4fd7568d" Oct 30 07:22:25.171333 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:25.171183 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa46f4d423c666abdd53e8f378dd057d6da1bb47ab561f24ada9032f4fd7568d"} err="failed to get container status \"aa46f4d423c666abdd53e8f378dd057d6da1bb47ab561f24ada9032f4fd7568d\": rpc error: code = NotFound desc = could not find container \"aa46f4d423c666abdd53e8f378dd057d6da1bb47ab561f24ada9032f4fd7568d\": container with ID starting with aa46f4d423c666abdd53e8f378dd057d6da1bb47ab561f24ada9032f4fd7568d not found: ID does not exist" Oct 30 07:22:25.180551 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:25.177606 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvvt2\" (UniqueName: \"kubernetes.io/projected/274fd0e0-2519-4369-8e34-341772f6a37d-kube-api-access-bvvt2\") pod \"payload-processing-e2e-ipp-b-597d57-6b875bfb88-trjgk\" (UID: \"274fd0e0-2519-4369-8e34-341772f6a37d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-6b875bfb88-trjgk" Oct 30 07:22:25.180551 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:25.177664 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/274fd0e0-2519-4369-8e34-341772f6a37d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-597d57-6b875bfb88-trjgk\" (UID: \"274fd0e0-2519-4369-8e34-341772f6a37d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-6b875bfb88-trjgk" Oct 30 07:22:25.180551 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:25.178730 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/274fd0e0-2519-4369-8e34-341772f6a37d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-597d57-6b875bfb88-trjgk\" (UID: \"274fd0e0-2519-4369-8e34-341772f6a37d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-6b875bfb88-trjgk" Oct 30 07:22:25.187294 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:25.186829 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-7cd9fbdcff-zn75b"] Oct 30 07:22:25.203812 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:25.203691 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvvt2\" (UniqueName: \"kubernetes.io/projected/274fd0e0-2519-4369-8e34-341772f6a37d-kube-api-access-bvvt2\") pod \"payload-processing-e2e-ipp-b-597d57-6b875bfb88-trjgk\" (UID: \"274fd0e0-2519-4369-8e34-341772f6a37d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-6b875bfb88-trjgk" Oct 30 07:22:25.216594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:25.216547 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bdd496-55f7c4fcdd-v4qkn"] Oct 30 07:22:25.216758 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:25.216649 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bdd496-55f7c4fcdd-v4qkn"] Oct 30 07:22:25.302268 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:25.302225 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-6b875bfb88-trjgk" Oct 30 07:22:25.393646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:25.392995 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-7fb879d9f5-p2sqj"] Oct 30 07:22:25.399166 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:22:25.397814 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d253264_032c_4c43_92d6_340d3abbdeee.slice/crio-7f0434f624f57767ca35cff40a25942c32213642f4e2771c34ba52a3f45db06b WatchSource:0}: Error finding container 7f0434f624f57767ca35cff40a25942c32213642f4e2771c34ba52a3f45db06b: Status 404 returned error can't find the container with id 7f0434f624f57767ca35cff40a25942c32213642f4e2771c34ba52a3f45db06b Oct 30 07:22:25.639855 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:22:25.639805 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod274fd0e0_2519_4369_8e34_341772f6a37d.slice/crio-fd91ef9c00c3147d710f6dc6bcf9137c05e0939aa8953a12cc05d9c3977a5929 WatchSource:0}: Error finding container fd91ef9c00c3147d710f6dc6bcf9137c05e0939aa8953a12cc05d9c3977a5929: Status 404 returned error can't find the container with id fd91ef9c00c3147d710f6dc6bcf9137c05e0939aa8953a12cc05d9c3977a5929 Oct 30 07:22:25.641614 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:25.641546 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-597d57-6b875bfb88-trjgk"] Oct 30 07:22:25.782636 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:25.780961 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20ccf7cf-223b-48aa-8845-af23df3d1c29" path="/var/lib/kubelet/pods/20ccf7cf-223b-48aa-8845-af23df3d1c29/volumes" Oct 30 07:22:25.782636 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:25.782399 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2898e6e0-5229-4c67-b3e3-8f2076284b86" path="/var/lib/kubelet/pods/2898e6e0-5229-4c67-b3e3-8f2076284b86/volumes" Oct 30 07:22:25.783072 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:25.783050 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d60f718d-86b8-4d75-a28a-7d7ad358cdb5" path="/var/lib/kubelet/pods/d60f718d-86b8-4d75-a28a-7d7ad358cdb5/volumes" Oct 30 07:22:26.078219 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:26.078120 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-6b875bfb88-trjgk" event={"ID":"274fd0e0-2519-4369-8e34-341772f6a37d","Type":"ContainerStarted","Data":"d23edbaba1ad19377932578f87e4afc0b56a5a5e0f84c14198b96d7b7a70f2d7"} Oct 30 07:22:26.078219 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:26.078183 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-6b875bfb88-trjgk" event={"ID":"274fd0e0-2519-4369-8e34-341772f6a37d","Type":"ContainerStarted","Data":"fd91ef9c00c3147d710f6dc6bcf9137c05e0939aa8953a12cc05d9c3977a5929"} Oct 30 07:22:26.078219 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:26.078209 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-6b875bfb88-trjgk" Oct 30 07:22:26.080789 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:26.080763 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-7fb879d9f5-p2sqj" event={"ID":"6d253264-032c-4c43-92d6-340d3abbdeee","Type":"ContainerStarted","Data":"340986529dd79d69cdde366f11e4263b88992418d865d07e20a47d443d4d2e3e"} Oct 30 07:22:26.080930 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:26.080815 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-7fb879d9f5-p2sqj" event={"ID":"6d253264-032c-4c43-92d6-340d3abbdeee","Type":"ContainerStarted","Data":"7f0434f624f57767ca35cff40a25942c32213642f4e2771c34ba52a3f45db06b"} Oct 30 07:22:26.080930 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:26.080837 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-7fb879d9f5-p2sqj" Oct 30 07:22:26.163081 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:26.163020 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-6b875bfb88-trjgk" podStartSLOduration=2.162998174 podStartE2EDuration="2.162998174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:22:26.159430536 +0000 UTC m=+1552.502478513" watchObservedRunningTime="2026-10-30 07:22:26.162998174 +0000 UTC m=+1552.506046141" Oct 30 07:22:26.260275 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:26.256379 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-7fb879d9f5-p2sqj" podStartSLOduration=2.256361235 podStartE2EDuration="2.256361235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:22:26.255239088 +0000 UTC m=+1552.598287054" watchObservedRunningTime="2026-10-30 07:22:26.256361235 +0000 UTC m=+1552.599409228" Oct 30 07:22:27.306877 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:27.306844 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp"] Oct 30 07:22:27.307357 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:27.307110 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" podUID="011a2449-9f58-427a-aa6e-89e4a911c2e5" containerName="istio-proxy" containerID="cri-o://7da231a53c77ced45e7f3c7b4ba8fb454a91b7e3a3689b0b4a1f986513ef9c51" gracePeriod=30 Oct 30 07:22:32.571724 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:32.571702 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:22:32.691610 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:32.691521 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/011a2449-9f58-427a-aa6e-89e4a911c2e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/011a2449-9f58-427a-aa6e-89e4a911c2e5-istio-podinfo\") pod \"011a2449-9f58-427a-aa6e-89e4a911c2e5\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " Oct 30 07:22:32.691610 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:32.691563 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-workload-certs\") pod \"011a2449-9f58-427a-aa6e-89e4a911c2e5\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " Oct 30 07:22:32.691899 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:32.691636 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-istio-envoy\") pod \"011a2449-9f58-427a-aa6e-89e4a911c2e5\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " Oct 30 07:22:32.691899 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:32.691670 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-istio-data\") pod \"011a2449-9f58-427a-aa6e-89e4a911c2e5\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " Oct 30 07:22:32.691899 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:32.691791 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-workload-socket\") pod \"011a2449-9f58-427a-aa6e-89e4a911c2e5\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " Oct 30 07:22:32.691899 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:32.691823 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/011a2449-9f58-427a-aa6e-89e4a911c2e5-istio-token\" (UniqueName: \"kubernetes.io/projected/011a2449-9f58-427a-aa6e-89e4a911c2e5-istio-token\") pod \"011a2449-9f58-427a-aa6e-89e4a911c2e5\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " Oct 30 07:22:32.692106 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:32.691903 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-credential-socket\") pod \"011a2449-9f58-427a-aa6e-89e4a911c2e5\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " Oct 30 07:22:32.692106 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:32.691964 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-workload-certs" pod "011a2449-9f58-427a-aa6e-89e4a911c2e5" (UID: "011a2449-9f58-427a-aa6e-89e4a911c2e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:22:32.692106 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:32.692013 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/011a2449-9f58-427a-aa6e-89e4a911c2e5-kube-api-access-pphq2\" (UniqueName: \"kubernetes.io/projected/011a2449-9f58-427a-aa6e-89e4a911c2e5-kube-api-access-pphq2\") pod \"011a2449-9f58-427a-aa6e-89e4a911c2e5\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " Oct 30 07:22:32.692245 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:32.692122 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/011a2449-9f58-427a-aa6e-89e4a911c2e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/011a2449-9f58-427a-aa6e-89e4a911c2e5-istiod-ca-cert\") pod \"011a2449-9f58-427a-aa6e-89e4a911c2e5\" (UID: \"011a2449-9f58-427a-aa6e-89e4a911c2e5\") " Oct 30 07:22:32.692245 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:32.692137 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-workload-socket" pod "011a2449-9f58-427a-aa6e-89e4a911c2e5" (UID: "011a2449-9f58-427a-aa6e-89e4a911c2e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:22:32.692245 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:32.692207 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-istio-data" pod "011a2449-9f58-427a-aa6e-89e4a911c2e5" (UID: "011a2449-9f58-427a-aa6e-89e4a911c2e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:22:32.692390 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:32.692265 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-credential-socket" pod "011a2449-9f58-427a-aa6e-89e4a911c2e5" (UID: "011a2449-9f58-427a-aa6e-89e4a911c2e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:22:32.692662 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:32.692642 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:32.692662 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:32.692654 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/011a2449-9f58-427a-aa6e-89e4a911c2e5-istiod-ca-cert" pod "011a2449-9f58-427a-aa6e-89e4a911c2e5" (UID: "011a2449-9f58-427a-aa6e-89e4a911c2e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:22:32.692819 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:32.692664 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:32.692819 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:32.692680 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:32.692819 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:32.692692 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:32.693987 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:32.693960 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-istio-envoy" pod "011a2449-9f58-427a-aa6e-89e4a911c2e5" (UID: "011a2449-9f58-427a-aa6e-89e4a911c2e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:22:32.694357 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:32.694314 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/011a2449-9f58-427a-aa6e-89e4a911c2e5-istio-token" pod "011a2449-9f58-427a-aa6e-89e4a911c2e5" (UID: "011a2449-9f58-427a-aa6e-89e4a911c2e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:22:32.694357 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:32.694318 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/011a2449-9f58-427a-aa6e-89e4a911c2e5-istio-podinfo" pod "011a2449-9f58-427a-aa6e-89e4a911c2e5" (UID: "011a2449-9f58-427a-aa6e-89e4a911c2e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:22:32.694476 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:32.694417 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/011a2449-9f58-427a-aa6e-89e4a911c2e5-kube-api-access-pphq2" pod "011a2449-9f58-427a-aa6e-89e4a911c2e5" (UID: "011a2449-9f58-427a-aa6e-89e4a911c2e5"). InnerVolumeSpecName "kube-api-access-pphq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:22:32.793934 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:32.793892 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pphq2\" (UniqueName: \"kubernetes.io/projected/011a2449-9f58-427a-aa6e-89e4a911c2e5-kube-api-access-pphq2\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:32.793934 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:32.793924 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/011a2449-9f58-427a-aa6e-89e4a911c2e5-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:32.793934 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:32.793938 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/011a2449-9f58-427a-aa6e-89e4a911c2e5-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:32.794194 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:32.793950 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/011a2449-9f58-427a-aa6e-89e4a911c2e5-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:32.794194 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:32.793962 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/011a2449-9f58-427a-aa6e-89e4a911c2e5-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:33.151588 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:33.151539 4951 generic.go:356] "Generic (PLEG): container finished" podID="011a2449-9f58-427a-aa6e-89e4a911c2e5" containerID="7da231a53c77ced45e7f3c7b4ba8fb454a91b7e3a3689b0b4a1f986513ef9c51" exitCode=0 Oct 30 07:22:33.151770 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:33.151625 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" event={"ID":"011a2449-9f58-427a-aa6e-89e4a911c2e5","Type":"ContainerDied","Data":"7da231a53c77ced45e7f3c7b4ba8fb454a91b7e3a3689b0b4a1f986513ef9c51"} Oct 30 07:22:33.151770 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:33.151634 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" Oct 30 07:22:33.151770 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:33.151718 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp" event={"ID":"011a2449-9f58-427a-aa6e-89e4a911c2e5","Type":"ContainerDied","Data":"c770f3549bb3776cfaaa547af1a23dc1144820c172579953056f24642181a162"} Oct 30 07:22:33.151770 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:33.151746 4951 scope.go:122] "RemoveContainer" containerID="7da231a53c77ced45e7f3c7b4ba8fb454a91b7e3a3689b0b4a1f986513ef9c51" Oct 30 07:22:33.163178 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:33.163142 4951 scope.go:122] "RemoveContainer" containerID="b8f3461c9a0c2a6751c03305a15a997e9dc65e63ebf891040e7c6c33e984cf2a" Oct 30 07:22:33.174310 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:33.174287 4951 scope.go:122] "RemoveContainer" containerID="7da231a53c77ced45e7f3c7b4ba8fb454a91b7e3a3689b0b4a1f986513ef9c51" Oct 30 07:22:33.174683 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:22:33.174659 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da231a53c77ced45e7f3c7b4ba8fb454a91b7e3a3689b0b4a1f986513ef9c51\": container with ID starting with 7da231a53c77ced45e7f3c7b4ba8fb454a91b7e3a3689b0b4a1f986513ef9c51 not found: ID does not exist" containerID="7da231a53c77ced45e7f3c7b4ba8fb454a91b7e3a3689b0b4a1f986513ef9c51" Oct 30 07:22:33.174762 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:33.174693 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da231a53c77ced45e7f3c7b4ba8fb454a91b7e3a3689b0b4a1f986513ef9c51"} err="failed to get container status \"7da231a53c77ced45e7f3c7b4ba8fb454a91b7e3a3689b0b4a1f986513ef9c51\": rpc error: code = NotFound desc = could not find container \"7da231a53c77ced45e7f3c7b4ba8fb454a91b7e3a3689b0b4a1f986513ef9c51\": container with ID starting with 7da231a53c77ced45e7f3c7b4ba8fb454a91b7e3a3689b0b4a1f986513ef9c51 not found: ID does not exist" Oct 30 07:22:33.174762 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:33.174714 4951 scope.go:122] "RemoveContainer" containerID="b8f3461c9a0c2a6751c03305a15a997e9dc65e63ebf891040e7c6c33e984cf2a" Oct 30 07:22:33.174998 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:22:33.174973 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8f3461c9a0c2a6751c03305a15a997e9dc65e63ebf891040e7c6c33e984cf2a\": container with ID starting with b8f3461c9a0c2a6751c03305a15a997e9dc65e63ebf891040e7c6c33e984cf2a not found: ID does not exist" containerID="b8f3461c9a0c2a6751c03305a15a997e9dc65e63ebf891040e7c6c33e984cf2a" Oct 30 07:22:33.175036 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:33.175002 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8f3461c9a0c2a6751c03305a15a997e9dc65e63ebf891040e7c6c33e984cf2a"} err="failed to get container status \"b8f3461c9a0c2a6751c03305a15a997e9dc65e63ebf891040e7c6c33e984cf2a\": rpc error: code = NotFound desc = could not find container \"b8f3461c9a0c2a6751c03305a15a997e9dc65e63ebf891040e7c6c33e984cf2a\": container with ID starting with b8f3461c9a0c2a6751c03305a15a997e9dc65e63ebf891040e7c6c33e984cf2a not found: ID does not exist" Oct 30 07:22:33.204070 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:33.204037 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp"] Oct 30 07:22:33.215928 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:33.215894 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-438855-openshift-default-5ffcc8795c-v46pp"] Oct 30 07:22:33.773176 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:33.773138 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="011a2449-9f58-427a-aa6e-89e4a911c2e5" path="/var/lib/kubelet/pods/011a2449-9f58-427a-aa6e-89e4a911c2e5/volumes" Oct 30 07:22:34.787493 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:34.787460 4951 scope.go:122] "RemoveContainer" containerID="05a08253454e72f3d45a4bf4e2e2297f5bc0d311cccf2f1865b4106dcd38704a" Oct 30 07:22:37.098987 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:37.098956 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-6b875bfb88-trjgk" Oct 30 07:22:37.099439 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:37.099155 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-7fb879d9f5-p2sqj" Oct 30 07:22:37.387067 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:37.386698 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn"] Oct 30 07:22:37.387067 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:37.386945 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn" podUID="2b02dc8b-0505-4f42-ad86-4301ddfbaf6e" containerName="maas-api" containerID="cri-o://b5d85a022dda9a624e686db2dc6d6b56e26c5caced738fd1f0b07d25d051b005" gracePeriod=30 Oct 30 07:22:37.708784 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:37.708751 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn" Oct 30 07:22:37.767469 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:37.767431 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b02dc8b-0505-4f42-ad86-4301ddfbaf6e-kube-api-access-ntsf8\" (UniqueName: \"kubernetes.io/projected/2b02dc8b-0505-4f42-ad86-4301ddfbaf6e-kube-api-access-ntsf8\") pod \"2b02dc8b-0505-4f42-ad86-4301ddfbaf6e\" (UID: \"2b02dc8b-0505-4f42-ad86-4301ddfbaf6e\") " Oct 30 07:22:37.767714 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:37.767511 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b02dc8b-0505-4f42-ad86-4301ddfbaf6e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b02dc8b-0505-4f42-ad86-4301ddfbaf6e-openshift-service-ca\") pod \"2b02dc8b-0505-4f42-ad86-4301ddfbaf6e\" (UID: \"2b02dc8b-0505-4f42-ad86-4301ddfbaf6e\") " Oct 30 07:22:37.767714 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:37.767634 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b02dc8b-0505-4f42-ad86-4301ddfbaf6e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b02dc8b-0505-4f42-ad86-4301ddfbaf6e-maas-api-tls\") pod \"2b02dc8b-0505-4f42-ad86-4301ddfbaf6e\" (UID: \"2b02dc8b-0505-4f42-ad86-4301ddfbaf6e\") " Oct 30 07:22:37.767714 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:37.767668 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b02dc8b-0505-4f42-ad86-4301ddfbaf6e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b02dc8b-0505-4f42-ad86-4301ddfbaf6e-metrics-certs\") pod \"2b02dc8b-0505-4f42-ad86-4301ddfbaf6e\" (UID: \"2b02dc8b-0505-4f42-ad86-4301ddfbaf6e\") " Oct 30 07:22:37.768374 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:37.768345 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b02dc8b-0505-4f42-ad86-4301ddfbaf6e-openshift-service-ca" pod "2b02dc8b-0505-4f42-ad86-4301ddfbaf6e" (UID: "2b02dc8b-0505-4f42-ad86-4301ddfbaf6e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:22:37.770615 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:37.770536 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b02dc8b-0505-4f42-ad86-4301ddfbaf6e-kube-api-access-ntsf8" pod "2b02dc8b-0505-4f42-ad86-4301ddfbaf6e" (UID: "2b02dc8b-0505-4f42-ad86-4301ddfbaf6e"). InnerVolumeSpecName "kube-api-access-ntsf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:22:37.770885 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:37.770861 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b02dc8b-0505-4f42-ad86-4301ddfbaf6e-metrics-certs" pod "2b02dc8b-0505-4f42-ad86-4301ddfbaf6e" (UID: "2b02dc8b-0505-4f42-ad86-4301ddfbaf6e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:22:37.771065 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:37.771033 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b02dc8b-0505-4f42-ad86-4301ddfbaf6e-maas-api-tls" pod "2b02dc8b-0505-4f42-ad86-4301ddfbaf6e" (UID: "2b02dc8b-0505-4f42-ad86-4301ddfbaf6e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:22:37.873600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:37.872707 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntsf8\" (UniqueName: \"kubernetes.io/projected/2b02dc8b-0505-4f42-ad86-4301ddfbaf6e-kube-api-access-ntsf8\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:37.873600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:37.872743 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b02dc8b-0505-4f42-ad86-4301ddfbaf6e-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:37.873600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:37.872758 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b02dc8b-0505-4f42-ad86-4301ddfbaf6e-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:37.873600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:37.872770 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b02dc8b-0505-4f42-ad86-4301ddfbaf6e-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:38.194291 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:38.194258 4951 generic.go:356] "Generic (PLEG): container finished" podID="2b02dc8b-0505-4f42-ad86-4301ddfbaf6e" containerID="b5d85a022dda9a624e686db2dc6d6b56e26c5caced738fd1f0b07d25d051b005" exitCode=0 Oct 30 07:22:38.194745 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:38.194303 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn" Oct 30 07:22:38.194745 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:38.194323 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn" event={"ID":"2b02dc8b-0505-4f42-ad86-4301ddfbaf6e","Type":"ContainerDied","Data":"b5d85a022dda9a624e686db2dc6d6b56e26c5caced738fd1f0b07d25d051b005"} Oct 30 07:22:38.194745 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:38.194366 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn" event={"ID":"2b02dc8b-0505-4f42-ad86-4301ddfbaf6e","Type":"ContainerDied","Data":"397cf6f75ecc1ff2f9e5745c7b191076c7e857794577bff3525c4a8c15a11141"} Oct 30 07:22:38.194745 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:38.194388 4951 scope.go:122] "RemoveContainer" containerID="b5d85a022dda9a624e686db2dc6d6b56e26c5caced738fd1f0b07d25d051b005" Oct 30 07:22:38.204731 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:38.204705 4951 scope.go:122] "RemoveContainer" containerID="b5d85a022dda9a624e686db2dc6d6b56e26c5caced738fd1f0b07d25d051b005" Oct 30 07:22:38.205029 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:22:38.205011 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5d85a022dda9a624e686db2dc6d6b56e26c5caced738fd1f0b07d25d051b005\": container with ID starting with b5d85a022dda9a624e686db2dc6d6b56e26c5caced738fd1f0b07d25d051b005 not found: ID does not exist" containerID="b5d85a022dda9a624e686db2dc6d6b56e26c5caced738fd1f0b07d25d051b005" Oct 30 07:22:38.205080 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:38.205046 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5d85a022dda9a624e686db2dc6d6b56e26c5caced738fd1f0b07d25d051b005"} err="failed to get container status \"b5d85a022dda9a624e686db2dc6d6b56e26c5caced738fd1f0b07d25d051b005\": rpc error: code = NotFound desc = could not find container \"b5d85a022dda9a624e686db2dc6d6b56e26c5caced738fd1f0b07d25d051b005\": container with ID starting with b5d85a022dda9a624e686db2dc6d6b56e26c5caced738fd1f0b07d25d051b005 not found: ID does not exist" Oct 30 07:22:38.288596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:38.285794 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn"] Oct 30 07:22:38.288596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:38.285857 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d2270-99597cc9f-p79dn"] Oct 30 07:22:39.808224 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:39.808166 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b02dc8b-0505-4f42-ad86-4301ddfbaf6e" path="/var/lib/kubelet/pods/2b02dc8b-0505-4f42-ad86-4301ddfbaf6e/volumes" Oct 30 07:22:40.091649 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.087230 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-68c9bfddc5-6vspl"] Oct 30 07:22:40.100594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.092928 4951 cpu_manager.go:414] "containerMap: removing container" podUID="2b02dc8b-0505-4f42-ad86-4301ddfbaf6e" containerName="maas-api" Oct 30 07:22:40.100835 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.100809 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b02dc8b-0505-4f42-ad86-4301ddfbaf6e" containerName="maas-api" Oct 30 07:22:40.101013 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.101001 4951 cpu_manager.go:414] "containerMap: removing container" podUID="011a2449-9f58-427a-aa6e-89e4a911c2e5" containerName="istio-proxy" Oct 30 07:22:40.101118 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.101105 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="011a2449-9f58-427a-aa6e-89e4a911c2e5" containerName="istio-proxy" Oct 30 07:22:40.101208 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.101198 4951 cpu_manager.go:414] "containerMap: removing container" podUID="011a2449-9f58-427a-aa6e-89e4a911c2e5" containerName="istio-proxy" Oct 30 07:22:40.101284 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.101273 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="011a2449-9f58-427a-aa6e-89e4a911c2e5" containerName="istio-proxy" Oct 30 07:22:40.101635 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.101615 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="011a2449-9f58-427a-aa6e-89e4a911c2e5" containerName="istio-proxy" Oct 30 07:22:40.105598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.101754 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="011a2449-9f58-427a-aa6e-89e4a911c2e5" containerName="istio-proxy" Oct 30 07:22:40.112693 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.112655 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b02dc8b-0505-4f42-ad86-4301ddfbaf6e" containerName="maas-api" Oct 30 07:22:40.127956 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.127888 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-68c9bfddc5-6vspl" Oct 30 07:22:40.160242 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.160206 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-68c9bfddc5-6vspl"] Oct 30 07:22:40.167058 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.167024 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-bdd496-dockercfg-fgm7v\"" Oct 30 07:22:40.180065 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.180034 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-bdd496\"" Oct 30 07:22:40.222807 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.222764 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bdd496-56b5b84df8-nh6gj"] Oct 30 07:22:40.239221 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.239173 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwg9z\" (UniqueName: \"kubernetes.io/projected/a5b46a24-504c-4849-ac82-2591ca1f0ce3-kube-api-access-nwg9z\") pod \"payload-pre-processing-e2e-ipp-b-bdd496-68c9bfddc5-6vspl\" (UID: \"a5b46a24-504c-4849-ac82-2591ca1f0ce3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-68c9bfddc5-6vspl" Oct 30 07:22:40.239411 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.239288 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5b46a24-504c-4849-ac82-2591ca1f0ce3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bdd496-68c9bfddc5-6vspl\" (UID: \"a5b46a24-504c-4849-ac82-2591ca1f0ce3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-68c9bfddc5-6vspl" Oct 30 07:22:40.243838 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.243807 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-56b5b84df8-nh6gj" Oct 30 07:22:40.249959 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.249771 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bdd496-56b5b84df8-nh6gj"] Oct 30 07:22:40.340478 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.340420 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwg9z\" (UniqueName: \"kubernetes.io/projected/a5b46a24-504c-4849-ac82-2591ca1f0ce3-kube-api-access-nwg9z\") pod \"payload-pre-processing-e2e-ipp-b-bdd496-68c9bfddc5-6vspl\" (UID: \"a5b46a24-504c-4849-ac82-2591ca1f0ce3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-68c9bfddc5-6vspl" Oct 30 07:22:40.340709 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.340508 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5b46a24-504c-4849-ac82-2591ca1f0ce3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bdd496-68c9bfddc5-6vspl\" (UID: \"a5b46a24-504c-4849-ac82-2591ca1f0ce3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-68c9bfddc5-6vspl" Oct 30 07:22:40.340709 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.340559 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24cd2f45-af16-4d34-b229-9b6b4c20d060-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bdd496-56b5b84df8-nh6gj\" (UID: \"24cd2f45-af16-4d34-b229-9b6b4c20d060\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-56b5b84df8-nh6gj" Oct 30 07:22:40.340709 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.340642 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxjs4\" (UniqueName: \"kubernetes.io/projected/24cd2f45-af16-4d34-b229-9b6b4c20d060-kube-api-access-jxjs4\") pod \"payload-processing-e2e-ipp-b-bdd496-56b5b84df8-nh6gj\" (UID: \"24cd2f45-af16-4d34-b229-9b6b4c20d060\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-56b5b84df8-nh6gj" Oct 30 07:22:40.341502 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.341471 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5b46a24-504c-4849-ac82-2591ca1f0ce3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bdd496-68c9bfddc5-6vspl\" (UID: \"a5b46a24-504c-4849-ac82-2591ca1f0ce3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-68c9bfddc5-6vspl" Oct 30 07:22:40.367333 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.367253 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwg9z\" (UniqueName: \"kubernetes.io/projected/a5b46a24-504c-4849-ac82-2591ca1f0ce3-kube-api-access-nwg9z\") pod \"payload-pre-processing-e2e-ipp-b-bdd496-68c9bfddc5-6vspl\" (UID: \"a5b46a24-504c-4849-ac82-2591ca1f0ce3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-68c9bfddc5-6vspl" Oct 30 07:22:40.442100 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.442057 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24cd2f45-af16-4d34-b229-9b6b4c20d060-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bdd496-56b5b84df8-nh6gj\" (UID: \"24cd2f45-af16-4d34-b229-9b6b4c20d060\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-56b5b84df8-nh6gj" Oct 30 07:22:40.442269 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.442111 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxjs4\" (UniqueName: \"kubernetes.io/projected/24cd2f45-af16-4d34-b229-9b6b4c20d060-kube-api-access-jxjs4\") pod \"payload-processing-e2e-ipp-b-bdd496-56b5b84df8-nh6gj\" (UID: \"24cd2f45-af16-4d34-b229-9b6b4c20d060\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-56b5b84df8-nh6gj" Oct 30 07:22:40.442866 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.442843 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24cd2f45-af16-4d34-b229-9b6b4c20d060-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bdd496-56b5b84df8-nh6gj\" (UID: \"24cd2f45-af16-4d34-b229-9b6b4c20d060\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-56b5b84df8-nh6gj" Oct 30 07:22:40.465215 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.465188 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxjs4\" (UniqueName: \"kubernetes.io/projected/24cd2f45-af16-4d34-b229-9b6b4c20d060-kube-api-access-jxjs4\") pod \"payload-processing-e2e-ipp-b-bdd496-56b5b84df8-nh6gj\" (UID: \"24cd2f45-af16-4d34-b229-9b6b4c20d060\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-56b5b84df8-nh6gj" Oct 30 07:22:40.498100 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.498067 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-68c9bfddc5-6vspl" Oct 30 07:22:40.567767 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.567418 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-56b5b84df8-nh6gj" Oct 30 07:22:40.618392 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.618306 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l"] Oct 30 07:22:40.626445 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.626416 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:22:40.644358 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.644331 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a8f428-openshift-default-dockercfg-h2t98\"" Oct 30 07:22:40.680718 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.680686 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l"] Oct 30 07:22:40.793638 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.789563 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-credential-socket\") pod \"e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:22:40.793638 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.789633 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03884451-d62c-4eea-827f-6e0c9359cbec-istio-token\") pod \"e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:22:40.793638 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.789697 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-istio-envoy\") pod \"e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:22:40.793638 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.789832 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-istio-data\") pod \"e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:22:40.793638 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.789874 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03884451-d62c-4eea-827f-6e0c9359cbec-istiod-ca-cert\") pod \"e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:22:40.793638 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.789909 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03884451-d62c-4eea-827f-6e0c9359cbec-istio-podinfo\") pod \"e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:22:40.793638 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.789937 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-workload-socket\") pod \"e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:22:40.793638 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.789976 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhps9\" (UniqueName: \"kubernetes.io/projected/03884451-d62c-4eea-827f-6e0c9359cbec-kube-api-access-fhps9\") pod \"e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:22:40.793638 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.790002 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-workload-certs\") pod \"e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:22:40.896659 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.893484 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-istio-envoy\") pod \"e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:22:40.929148 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.929030 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-68c9bfddc5-6vspl"] Oct 30 07:22:40.929148 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.929078 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-istio-envoy\") pod \"e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:22:40.938630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.930315 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-istio-data\") pod \"e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:22:40.938630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.930385 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03884451-d62c-4eea-827f-6e0c9359cbec-istiod-ca-cert\") pod \"e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:22:40.938630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.930427 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03884451-d62c-4eea-827f-6e0c9359cbec-istio-podinfo\") pod \"e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:22:40.938630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.930479 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-workload-socket\") pod \"e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:22:40.938630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.930609 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhps9\" (UniqueName: \"kubernetes.io/projected/03884451-d62c-4eea-827f-6e0c9359cbec-kube-api-access-fhps9\") pod \"e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:22:40.938630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.930646 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-workload-certs\") pod \"e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:22:40.938630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.930777 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-credential-socket\") pod \"e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:22:40.938630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.930804 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03884451-d62c-4eea-827f-6e0c9359cbec-istio-token\") pod \"e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:22:40.938630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.931472 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-istio-data\") pod \"e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:22:40.938630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.931637 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-workload-socket\") pod \"e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:22:40.938630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.931953 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-workload-certs\") pod \"e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:22:40.938630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.932176 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-credential-socket\") pod \"e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:22:40.938630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.932363 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03884451-d62c-4eea-827f-6e0c9359cbec-istiod-ca-cert\") pod \"e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:22:40.941942 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.941378 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03884451-d62c-4eea-827f-6e0c9359cbec-istio-podinfo\") pod \"e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:22:40.973066 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:40.963910 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhps9\" (UniqueName: \"kubernetes.io/projected/03884451-d62c-4eea-827f-6e0c9359cbec-kube-api-access-fhps9\") pod \"e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:22:41.015966 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:41.015888 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03884451-d62c-4eea-827f-6e0c9359cbec-istio-token\") pod \"e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:22:41.137620 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:41.137565 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bdd496-56b5b84df8-nh6gj"] Oct 30 07:22:41.138287 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:22:41.138262 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24cd2f45_af16_4d34_b229_9b6b4c20d060.slice/crio-afd26f7765b949cc2217fabefbdc40bb35848aea33f1c3f6a41d818986a0ea64 WatchSource:0}: Error finding container afd26f7765b949cc2217fabefbdc40bb35848aea33f1c3f6a41d818986a0ea64: Status 404 returned error can't find the container with id afd26f7765b949cc2217fabefbdc40bb35848aea33f1c3f6a41d818986a0ea64 Oct 30 07:22:41.221076 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:41.221048 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-56b5b84df8-nh6gj" event={"ID":"24cd2f45-af16-4d34-b229-9b6b4c20d060","Type":"ContainerStarted","Data":"54ad81fb4c272c6f58f9925b709c01f85f42f568acbbe14e9c813bc19349a03d"} Oct 30 07:22:41.221212 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:41.221114 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-56b5b84df8-nh6gj" Oct 30 07:22:41.221212 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:41.221140 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-56b5b84df8-nh6gj" event={"ID":"24cd2f45-af16-4d34-b229-9b6b4c20d060","Type":"ContainerStarted","Data":"afd26f7765b949cc2217fabefbdc40bb35848aea33f1c3f6a41d818986a0ea64"} Oct 30 07:22:41.222975 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:41.222951 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-68c9bfddc5-6vspl" event={"ID":"a5b46a24-504c-4849-ac82-2591ca1f0ce3","Type":"ContainerStarted","Data":"fb91dbce3eeb785099b66bc0e3305d7b3a9e533770c1b875adf74c65abd4139e"} Oct 30 07:22:41.223068 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:41.222996 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-68c9bfddc5-6vspl" event={"ID":"a5b46a24-504c-4849-ac82-2591ca1f0ce3","Type":"ContainerStarted","Data":"dbcb2c6712c6d45056af7167e1b4427f6503629ce50df7f635b7f73f692a2977"} Oct 30 07:22:41.223202 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:41.223179 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-68c9bfddc5-6vspl" Oct 30 07:22:41.248449 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:41.248409 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:22:41.266782 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:41.266737 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-56b5b84df8-nh6gj" podStartSLOduration=1.266722323 podStartE2EDuration="1.266722323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:22:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:22:41.262355162 +0000 UTC m=+1567.605403127" watchObservedRunningTime="2026-10-30 07:22:41.266722323 +0000 UTC m=+1567.609770287" Oct 30 07:22:41.306910 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:41.306838 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-68c9bfddc5-6vspl" podStartSLOduration=1.306817485 podStartE2EDuration="1.306817485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:22:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:22:41.302878084 +0000 UTC m=+1567.645926052" watchObservedRunningTime="2026-10-30 07:22:41.306817485 +0000 UTC m=+1567.649865451" Oct 30 07:22:41.604177 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:22:41.604134 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03884451_d62c_4eea_827f_6e0c9359cbec.slice/crio-c84a6c49df830564def2ca43b8e4bae666c8659b1b8be13430a43279500aeee1 WatchSource:0}: Error finding container c84a6c49df830564def2ca43b8e4bae666c8659b1b8be13430a43279500aeee1: Status 404 returned error can't find the container with id c84a6c49df830564def2ca43b8e4bae666c8659b1b8be13430a43279500aeee1 Oct 30 07:22:41.604177 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:41.604151 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l"] Oct 30 07:22:41.607416 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:41.607373 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:22:41.607513 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:41.607461 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:22:41.607513 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:41.607501 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:22:42.233285 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:42.233243 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" event={"ID":"03884451-d62c-4eea-827f-6e0c9359cbec","Type":"ContainerStarted","Data":"9c409058287aed26d6bcb72f9661a2627c4114bc51e907727cba31aca4eab50c"} Oct 30 07:22:42.233836 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:42.233312 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" event={"ID":"03884451-d62c-4eea-827f-6e0c9359cbec","Type":"ContainerStarted","Data":"c84a6c49df830564def2ca43b8e4bae666c8659b1b8be13430a43279500aeee1"} Oct 30 07:22:42.249516 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:42.249476 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:22:42.256878 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:42.256835 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:22:42.308495 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:42.308445 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" podStartSLOduration=2.308431078 podStartE2EDuration="2.308431078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:22:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:22:42.293733322 +0000 UTC m=+1568.636781311" watchObservedRunningTime="2026-10-30 07:22:42.308431078 +0000 UTC m=+1568.651479123" Oct 30 07:22:43.243032 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:43.242989 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:22:43.244586 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:43.244542 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:22:45.142411 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:45.142375 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l"] Oct 30 07:22:45.143011 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:45.142675 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" podUID="9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b" containerName="istio-proxy" containerID="cri-o://410542a5625835bd790467d2621db81aad747f62a19df3743a09aa8d92c69071" gracePeriod=30 Oct 30 07:22:47.164383 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:47.164202 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp"] Oct 30 07:22:47.183924 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:47.183886 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:22:47.202017 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:47.200594 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp"] Oct 30 07:22:47.206658 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:47.206629 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-96ca87-openshift-default-dockercfg-snfmn\"" Oct 30 07:22:47.264706 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:47.264662 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istio-data\") pod \"e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:22:47.264900 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:47.264872 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istiod-ca-cert\") pod \"e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:22:47.264962 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:47.264926 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-workload-socket\") pod \"e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:22:47.265024 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:47.265002 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-workload-certs\") pod \"e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:22:47.265135 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:47.265122 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istio-podinfo\") pod \"e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:22:47.265185 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:47.265155 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istio-envoy\") pod \"e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:22:47.265185 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:47.265177 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8xjf\" (UniqueName: \"kubernetes.io/projected/d5e6118a-e82b-4c28-b716-8e6cbaa81474-kube-api-access-r8xjf\") pod \"e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:22:47.265275 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:47.265220 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-credential-socket\") pod \"e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:22:47.265313 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:47.265292 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istio-token\") pod \"e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:22:47.400267 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:47.400230 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istiod-ca-cert\") pod \"e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:22:47.401327 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:47.401303 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-workload-socket\") pod \"e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:22:47.401865 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:47.401847 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-workload-certs\") pod \"e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:22:47.402307 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:47.402284 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istio-podinfo\") pod \"e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:22:47.402892 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:47.402873 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istio-envoy\") pod \"e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:22:47.465061 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:47.464958 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8xjf\" (UniqueName: \"kubernetes.io/projected/d5e6118a-e82b-4c28-b716-8e6cbaa81474-kube-api-access-r8xjf\") pod \"e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:22:47.465742 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:47.465703 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-credential-socket\") pod \"e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:22:47.466018 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:47.465996 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istio-token\") pod \"e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:22:47.466242 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:47.466226 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istio-data\") pod \"e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:22:47.466613 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:47.401226 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istiod-ca-cert\") pod \"e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:22:47.466761 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:47.402185 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-workload-certs\") pod \"e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:22:47.466854 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:47.401800 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-workload-socket\") pod \"e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:22:47.467287 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:47.467265 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-credential-socket\") pod \"e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:22:47.467438 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:47.415908 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istio-envoy\") pod \"e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:22:47.467804 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:47.444139 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istio-podinfo\") pod \"e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:22:47.468218 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:47.468198 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istio-data\") pod \"e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:22:47.508841 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:47.508806 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8xjf\" (UniqueName: \"kubernetes.io/projected/d5e6118a-e82b-4c28-b716-8e6cbaa81474-kube-api-access-r8xjf\") pod \"e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:22:47.529562 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:47.529528 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istio-token\") pod \"e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:22:47.813949 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:47.813917 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:22:48.165083 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:48.164981 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp"] Oct 30 07:22:48.170667 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:22:48.170632 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e6118a_e82b_4c28_b716_8e6cbaa81474.slice/crio-7f7e39f9669494a694d2d99c07e674500610f069b65c82c21e7a6c4617f14be5 WatchSource:0}: Error finding container 7f7e39f9669494a694d2d99c07e674500610f069b65c82c21e7a6c4617f14be5: Status 404 returned error can't find the container with id 7f7e39f9669494a694d2d99c07e674500610f069b65c82c21e7a6c4617f14be5 Oct 30 07:22:48.174321 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:48.174280 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:22:48.174432 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:48.174379 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:22:48.174432 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:48.174412 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:22:48.457705 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:48.457562 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" event={"ID":"d5e6118a-e82b-4c28-b716-8e6cbaa81474","Type":"ContainerStarted","Data":"37f97d4e834152adcfa2b17a2477915d3bd49fafd25c081b312b763b8cd8905b"} Oct 30 07:22:48.457705 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:48.457669 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" event={"ID":"d5e6118a-e82b-4c28-b716-8e6cbaa81474","Type":"ContainerStarted","Data":"7f7e39f9669494a694d2d99c07e674500610f069b65c82c21e7a6c4617f14be5"} Oct 30 07:22:48.504018 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:48.503959 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" podStartSLOduration=1.5039411029999998 podStartE2EDuration="1.503941103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:22:48.502284559 +0000 UTC m=+1574.845332524" watchObservedRunningTime="2026-10-30 07:22:48.503941103 +0000 UTC m=+1574.846989070" Oct 30 07:22:48.814680 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:48.814646 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:22:49.201874 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:49.201782 4951 patch_prober.go:29] interesting pod/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:22:49.201874 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:49.201843 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" podUID="9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:22:49.820423 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:49.820391 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:22:50.467168 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.467137 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:22:50.475072 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.475042 4951 generic.go:356] "Generic (PLEG): container finished" podID="9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b" containerID="410542a5625835bd790467d2621db81aad747f62a19df3743a09aa8d92c69071" exitCode=0 Oct 30 07:22:50.475210 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.475085 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" Oct 30 07:22:50.475210 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.475125 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" event={"ID":"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b","Type":"ContainerDied","Data":"410542a5625835bd790467d2621db81aad747f62a19df3743a09aa8d92c69071"} Oct 30 07:22:50.475210 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.475191 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l" event={"ID":"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b","Type":"ContainerDied","Data":"09dc38f5ea51b6b51df8f2faf57cbb5b7ddc62b4e367d4df9a78c6eaed6b6a19"} Oct 30 07:22:50.475315 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.475212 4951 scope.go:122] "RemoveContainer" containerID="410542a5625835bd790467d2621db81aad747f62a19df3743a09aa8d92c69071" Oct 30 07:22:50.475442 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.475415 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:22:50.476484 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.476465 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:22:50.488841 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.488808 4951 scope.go:122] "RemoveContainer" containerID="893fe6868772fd31a178a7746a5974b29097011e48dbc7c66004b9e385ce56e5" Oct 30 07:22:50.499900 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.499878 4951 scope.go:122] "RemoveContainer" containerID="410542a5625835bd790467d2621db81aad747f62a19df3743a09aa8d92c69071" Oct 30 07:22:50.500250 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:22:50.500224 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"410542a5625835bd790467d2621db81aad747f62a19df3743a09aa8d92c69071\": container with ID starting with 410542a5625835bd790467d2621db81aad747f62a19df3743a09aa8d92c69071 not found: ID does not exist" containerID="410542a5625835bd790467d2621db81aad747f62a19df3743a09aa8d92c69071" Oct 30 07:22:50.500293 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.500270 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"410542a5625835bd790467d2621db81aad747f62a19df3743a09aa8d92c69071"} err="failed to get container status \"410542a5625835bd790467d2621db81aad747f62a19df3743a09aa8d92c69071\": rpc error: code = NotFound desc = could not find container \"410542a5625835bd790467d2621db81aad747f62a19df3743a09aa8d92c69071\": container with ID starting with 410542a5625835bd790467d2621db81aad747f62a19df3743a09aa8d92c69071 not found: ID does not exist" Oct 30 07:22:50.500340 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.500299 4951 scope.go:122] "RemoveContainer" containerID="893fe6868772fd31a178a7746a5974b29097011e48dbc7c66004b9e385ce56e5" Oct 30 07:22:50.500630 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:22:50.500608 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"893fe6868772fd31a178a7746a5974b29097011e48dbc7c66004b9e385ce56e5\": container with ID starting with 893fe6868772fd31a178a7746a5974b29097011e48dbc7c66004b9e385ce56e5 not found: ID does not exist" containerID="893fe6868772fd31a178a7746a5974b29097011e48dbc7c66004b9e385ce56e5" Oct 30 07:22:50.500722 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.500643 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"893fe6868772fd31a178a7746a5974b29097011e48dbc7c66004b9e385ce56e5"} err="failed to get container status \"893fe6868772fd31a178a7746a5974b29097011e48dbc7c66004b9e385ce56e5\": rpc error: code = NotFound desc = could not find container \"893fe6868772fd31a178a7746a5974b29097011e48dbc7c66004b9e385ce56e5\": container with ID starting with 893fe6868772fd31a178a7746a5974b29097011e48dbc7c66004b9e385ce56e5 not found: ID does not exist" Oct 30 07:22:50.629595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.629537 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-workload-certs\") pod \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " Oct 30 07:22:50.629790 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.629657 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-kube-api-access-pwc6q\" (UniqueName: \"kubernetes.io/projected/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-kube-api-access-pwc6q\") pod \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " Oct 30 07:22:50.629790 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.629704 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istio-token\" (UniqueName: \"kubernetes.io/projected/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istio-token\") pod \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " Oct 30 07:22:50.629790 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.629732 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istio-data\") pod \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " Oct 30 07:22:50.629953 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.629801 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istio-envoy\") pod \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " Oct 30 07:22:50.629953 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.629883 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istio-podinfo\") pod \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " Oct 30 07:22:50.629953 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.629939 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-credential-socket\") pod \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " Oct 30 07:22:50.630172 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.630149 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-workload-socket\") pod \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " Oct 30 07:22:50.630245 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.630217 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istiod-ca-cert\") pod \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\" (UID: \"9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b\") " Oct 30 07:22:50.643623 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.629876 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-workload-certs" pod "9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b" (UID: "9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:22:50.643623 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.630072 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istio-data" pod "9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b" (UID: "9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:22:50.643623 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.634802 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istio-envoy" pod "9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b" (UID: "9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:22:50.643623 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.638050 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istiod-ca-cert" pod "9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b" (UID: "9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:22:50.643623 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.638359 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-credential-socket" pod "9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b" (UID: "9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:22:50.643623 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.638537 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-workload-socket" pod "9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b" (UID: "9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:22:50.643623 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.640079 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-kube-api-access-pwc6q" pod "9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b" (UID: "9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b"). InnerVolumeSpecName "kube-api-access-pwc6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:22:50.643623 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.641661 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istio-podinfo" pod "9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b" (UID: "9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:22:50.647100 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.647067 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istio-token" pod "9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b" (UID: "9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:22:50.732509 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.732415 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:50.732509 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.732456 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:50.732509 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.732472 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwc6q\" (UniqueName: \"kubernetes.io/projected/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-kube-api-access-pwc6q\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:50.732509 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.732487 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:50.732509 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.732501 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:50.732509 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.732513 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:50.732876 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.732526 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:50.732876 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.732538 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:50.732876 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.732552 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:50.822188 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.822160 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l"] Oct 30 07:22:50.826380 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:50.826353 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-3d2270-openshift-default-6d66858489-mlf7l"] Oct 30 07:22:51.639361 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:51.639324 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efce7b-86c7d47c8d-8tvds"] Oct 30 07:22:51.639901 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:51.639590 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-86c7d47c8d-8tvds" podUID="f4d71ae8-3979-48f6-ab97-612b599dd6c4" containerName="payload-processing" containerID="cri-o://0f93525973df6f1ceea43366b36a593d4eba4287dcde1cfd73989d7d78ac7454" gracePeriod=30 Oct 30 07:22:51.775241 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:51.775208 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b" path="/var/lib/kubelet/pods/9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b/volumes" Oct 30 07:22:51.928506 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:51.928478 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-86c7d47c8d-8tvds" Oct 30 07:22:51.949259 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:51.949228 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4d71ae8-3979-48f6-ab97-612b599dd6c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4d71ae8-3979-48f6-ab97-612b599dd6c4-plugins-config-volume\") pod \"f4d71ae8-3979-48f6-ab97-612b599dd6c4\" (UID: \"f4d71ae8-3979-48f6-ab97-612b599dd6c4\") " Oct 30 07:22:51.949411 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:51.949344 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4d71ae8-3979-48f6-ab97-612b599dd6c4-kube-api-access-qd6mx\" (UniqueName: \"kubernetes.io/projected/f4d71ae8-3979-48f6-ab97-612b599dd6c4-kube-api-access-qd6mx\") pod \"f4d71ae8-3979-48f6-ab97-612b599dd6c4\" (UID: \"f4d71ae8-3979-48f6-ab97-612b599dd6c4\") " Oct 30 07:22:51.949603 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:51.949566 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4d71ae8-3979-48f6-ab97-612b599dd6c4-plugins-config-volume" pod "f4d71ae8-3979-48f6-ab97-612b599dd6c4" (UID: "f4d71ae8-3979-48f6-ab97-612b599dd6c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:22:51.949903 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:51.949856 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4d71ae8-3979-48f6-ab97-612b599dd6c4-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:51.951896 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:51.951871 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4d71ae8-3979-48f6-ab97-612b599dd6c4-kube-api-access-qd6mx" pod "f4d71ae8-3979-48f6-ab97-612b599dd6c4" (UID: "f4d71ae8-3979-48f6-ab97-612b599dd6c4"). InnerVolumeSpecName "kube-api-access-qd6mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:22:52.138138 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:52.138083 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd6mx\" (UniqueName: \"kubernetes.io/projected/f4d71ae8-3979-48f6-ab97-612b599dd6c4-kube-api-access-qd6mx\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:22:52.236222 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:52.236129 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-56b5b84df8-nh6gj" Oct 30 07:22:52.236747 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:52.236722 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-68c9bfddc5-6vspl" Oct 30 07:22:52.507448 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:52.506743 4951 generic.go:356] "Generic (PLEG): container finished" podID="f4d71ae8-3979-48f6-ab97-612b599dd6c4" containerID="0f93525973df6f1ceea43366b36a593d4eba4287dcde1cfd73989d7d78ac7454" exitCode=0 Oct 30 07:22:52.507448 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:52.506796 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-86c7d47c8d-8tvds" Oct 30 07:22:52.507448 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:52.506821 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-86c7d47c8d-8tvds" event={"ID":"f4d71ae8-3979-48f6-ab97-612b599dd6c4","Type":"ContainerDied","Data":"0f93525973df6f1ceea43366b36a593d4eba4287dcde1cfd73989d7d78ac7454"} Oct 30 07:22:52.507448 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:52.506906 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-86c7d47c8d-8tvds" event={"ID":"f4d71ae8-3979-48f6-ab97-612b599dd6c4","Type":"ContainerDied","Data":"e5002a983d251ccb9985f057121d1defd286366e5a01b7f0ed51ccbfd79f568b"} Oct 30 07:22:52.507448 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:52.506965 4951 scope.go:122] "RemoveContainer" containerID="0f93525973df6f1ceea43366b36a593d4eba4287dcde1cfd73989d7d78ac7454" Oct 30 07:22:52.520194 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:52.520167 4951 scope.go:122] "RemoveContainer" containerID="0f93525973df6f1ceea43366b36a593d4eba4287dcde1cfd73989d7d78ac7454" Oct 30 07:22:52.520649 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:22:52.520623 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f93525973df6f1ceea43366b36a593d4eba4287dcde1cfd73989d7d78ac7454\": container with ID starting with 0f93525973df6f1ceea43366b36a593d4eba4287dcde1cfd73989d7d78ac7454 not found: ID does not exist" containerID="0f93525973df6f1ceea43366b36a593d4eba4287dcde1cfd73989d7d78ac7454" Oct 30 07:22:52.520773 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:52.520657 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f93525973df6f1ceea43366b36a593d4eba4287dcde1cfd73989d7d78ac7454"} err="failed to get container status \"0f93525973df6f1ceea43366b36a593d4eba4287dcde1cfd73989d7d78ac7454\": rpc error: code = NotFound desc = could not find container \"0f93525973df6f1ceea43366b36a593d4eba4287dcde1cfd73989d7d78ac7454\": container with ID starting with 0f93525973df6f1ceea43366b36a593d4eba4287dcde1cfd73989d7d78ac7454 not found: ID does not exist" Oct 30 07:22:52.561341 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:52.561304 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efce7b-86c7d47c8d-8tvds"] Oct 30 07:22:52.568432 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:52.568401 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efce7b-86c7d47c8d-8tvds"] Oct 30 07:22:53.106599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:53.098948 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-96ca87-65d6cb4bdd-clzvf"] Oct 30 07:22:53.106599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:53.100022 4951 cpu_manager.go:414] "containerMap: removing container" podUID="f4d71ae8-3979-48f6-ab97-612b599dd6c4" containerName="payload-processing" Oct 30 07:22:53.106599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:53.100044 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4d71ae8-3979-48f6-ab97-612b599dd6c4" containerName="payload-processing" Oct 30 07:22:53.106599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:53.100076 4951 cpu_manager.go:414] "containerMap: removing container" podUID="9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b" containerName="istio-proxy" Oct 30 07:22:53.106599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:53.100085 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b" containerName="istio-proxy" Oct 30 07:22:53.106599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:53.100114 4951 cpu_manager.go:414] "containerMap: removing container" podUID="9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b" containerName="istio-proxy" Oct 30 07:22:53.106599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:53.100122 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b" containerName="istio-proxy" Oct 30 07:22:53.106599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:53.100403 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4d71ae8-3979-48f6-ab97-612b599dd6c4" containerName="payload-processing" Oct 30 07:22:53.106599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:53.100426 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b" containerName="istio-proxy" Oct 30 07:22:53.106599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:53.100439 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c60baf9-8da4-4ade-8ec6-f6a10a6aae4b" containerName="istio-proxy" Oct 30 07:22:53.132335 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:53.124076 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-96ca87-65d6cb4bdd-clzvf"] Oct 30 07:22:53.132335 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:53.124074 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-96ca87-65d6cb4bdd-clzvf" Oct 30 07:22:53.153607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:53.149547 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-96ca87\"" Oct 30 07:22:53.153607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:53.149773 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-96ca87-dockercfg-r9q79\"" Oct 30 07:22:53.205624 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:53.202036 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffhjf\" (UniqueName: \"kubernetes.io/projected/41a7e771-8212-4749-a905-4826e1814eff-kube-api-access-ffhjf\") pod \"payload-processing-e2e-praxis-a-96ca87-65d6cb4bdd-clzvf\" (UID: \"41a7e771-8212-4749-a905-4826e1814eff\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-96ca87-65d6cb4bdd-clzvf" Oct 30 07:22:53.205624 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:53.202242 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41a7e771-8212-4749-a905-4826e1814eff-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-96ca87-65d6cb4bdd-clzvf\" (UID: \"41a7e771-8212-4749-a905-4826e1814eff\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-96ca87-65d6cb4bdd-clzvf" Oct 30 07:22:53.278780 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:22:53.278656 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:22:53.303286 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:53.303242 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41a7e771-8212-4749-a905-4826e1814eff-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-96ca87-65d6cb4bdd-clzvf\" (UID: \"41a7e771-8212-4749-a905-4826e1814eff\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-96ca87-65d6cb4bdd-clzvf" Oct 30 07:22:53.304527 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:53.304489 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffhjf\" (UniqueName: \"kubernetes.io/projected/41a7e771-8212-4749-a905-4826e1814eff-kube-api-access-ffhjf\") pod \"payload-processing-e2e-praxis-a-96ca87-65d6cb4bdd-clzvf\" (UID: \"41a7e771-8212-4749-a905-4826e1814eff\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-96ca87-65d6cb4bdd-clzvf" Oct 30 07:22:53.309504 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:53.305843 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41a7e771-8212-4749-a905-4826e1814eff-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-96ca87-65d6cb4bdd-clzvf\" (UID: \"41a7e771-8212-4749-a905-4826e1814eff\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-96ca87-65d6cb4bdd-clzvf" Oct 30 07:22:53.341196 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:53.341156 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffhjf\" (UniqueName: \"kubernetes.io/projected/41a7e771-8212-4749-a905-4826e1814eff-kube-api-access-ffhjf\") pod \"payload-processing-e2e-praxis-a-96ca87-65d6cb4bdd-clzvf\" (UID: \"41a7e771-8212-4749-a905-4826e1814eff\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-96ca87-65d6cb4bdd-clzvf" Oct 30 07:22:53.493814 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:53.493714 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-96ca87-65d6cb4bdd-clzvf" Oct 30 07:22:53.786840 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:53.786807 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4d71ae8-3979-48f6-ab97-612b599dd6c4" path="/var/lib/kubelet/pods/f4d71ae8-3979-48f6-ab97-612b599dd6c4/volumes" Oct 30 07:22:53.805981 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:53.805952 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-96ca87-65d6cb4bdd-clzvf"] Oct 30 07:22:53.807168 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:22:53.807141 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a7e771_8212_4749_a905_4826e1814eff.slice/crio-d5c1b1f555ce52f90c6d4720634cbe9d1163f950a627228f5e675926902995c1 WatchSource:0}: Error finding container d5c1b1f555ce52f90c6d4720634cbe9d1163f950a627228f5e675926902995c1: Status 404 returned error can't find the container with id d5c1b1f555ce52f90c6d4720634cbe9d1163f950a627228f5e675926902995c1 Oct 30 07:22:54.527400 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:54.527366 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-96ca87-65d6cb4bdd-clzvf" event={"ID":"41a7e771-8212-4749-a905-4826e1814eff","Type":"ContainerStarted","Data":"4a1cfcd5866f05cb19e0df73c4004c1ae02f096c3a218971dc3fd9b6077b51e5"} Oct 30 07:22:54.527881 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:54.527425 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-96ca87-65d6cb4bdd-clzvf" event={"ID":"41a7e771-8212-4749-a905-4826e1814eff","Type":"ContainerStarted","Data":"d5c1b1f555ce52f90c6d4720634cbe9d1163f950a627228f5e675926902995c1"} Oct 30 07:22:54.527881 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:54.527594 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-96ca87-65d6cb4bdd-clzvf" Oct 30 07:22:54.568920 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:22:54.568856 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-96ca87-65d6cb4bdd-clzvf" podStartSLOduration=1.56883704 podStartE2EDuration="1.56883704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:22:54.566865026 +0000 UTC m=+1580.909912991" watchObservedRunningTime="2026-10-30 07:22:54.56883704 +0000 UTC m=+1580.911885005" Oct 30 07:23:00.667771 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:00.667734 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn"] Oct 30 07:23:00.668544 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:00.668509 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" podUID="5c1b682b-01eb-4717-9184-d81851ade4b0" containerName="istio-proxy" containerID="cri-o://3e52bfcd6536d75017df91674af8f7417a7a20b386b9711fe8361a4bd8676c15" gracePeriod=30 Oct 30 07:23:04.067435 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:04.067388 4951 patch_prober.go:29] interesting pod/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:23:04.068182 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:04.068134 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" podUID="5c1b682b-01eb-4717-9184-d81851ade4b0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:23:05.558355 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:05.558323 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-96ca87-65d6cb4bdd-clzvf" Oct 30 07:23:05.990508 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:05.990467 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:23:06.100886 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.100847 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-workload-certs\") pod \"5c1b682b-01eb-4717-9184-d81851ade4b0\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " Oct 30 07:23:06.101085 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.100900 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c1b682b-01eb-4717-9184-d81851ade4b0-istio-token\" (UniqueName: \"kubernetes.io/projected/5c1b682b-01eb-4717-9184-d81851ade4b0-istio-token\") pod \"5c1b682b-01eb-4717-9184-d81851ade4b0\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " Oct 30 07:23:06.101085 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.100973 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c1b682b-01eb-4717-9184-d81851ade4b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c1b682b-01eb-4717-9184-d81851ade4b0-istiod-ca-cert\") pod \"5c1b682b-01eb-4717-9184-d81851ade4b0\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " Oct 30 07:23:06.101085 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.101042 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-istio-envoy\") pod \"5c1b682b-01eb-4717-9184-d81851ade4b0\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " Oct 30 07:23:06.101263 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.101102 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c1b682b-01eb-4717-9184-d81851ade4b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c1b682b-01eb-4717-9184-d81851ade4b0-istio-podinfo\") pod \"5c1b682b-01eb-4717-9184-d81851ade4b0\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " Oct 30 07:23:06.101263 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.101184 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-workload-certs" pod "5c1b682b-01eb-4717-9184-d81851ade4b0" (UID: "5c1b682b-01eb-4717-9184-d81851ade4b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:06.101383 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.101269 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-credential-socket\") pod \"5c1b682b-01eb-4717-9184-d81851ade4b0\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " Oct 30 07:23:06.101383 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.101344 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-workload-socket\") pod \"5c1b682b-01eb-4717-9184-d81851ade4b0\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " Oct 30 07:23:06.101495 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.101388 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c1b682b-01eb-4717-9184-d81851ade4b0-kube-api-access-nvctj\" (UniqueName: \"kubernetes.io/projected/5c1b682b-01eb-4717-9184-d81851ade4b0-kube-api-access-nvctj\") pod \"5c1b682b-01eb-4717-9184-d81851ade4b0\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " Oct 30 07:23:06.101495 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.101469 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-istio-data\") pod \"5c1b682b-01eb-4717-9184-d81851ade4b0\" (UID: \"5c1b682b-01eb-4717-9184-d81851ade4b0\") " Oct 30 07:23:06.101702 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.101530 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c1b682b-01eb-4717-9184-d81851ade4b0-istiod-ca-cert" pod "5c1b682b-01eb-4717-9184-d81851ade4b0" (UID: "5c1b682b-01eb-4717-9184-d81851ade4b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:23:06.101820 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.101801 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-credential-socket" pod "5c1b682b-01eb-4717-9184-d81851ade4b0" (UID: "5c1b682b-01eb-4717-9184-d81851ade4b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:06.102287 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.102112 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:06.102287 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.102139 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c1b682b-01eb-4717-9184-d81851ade4b0-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:06.102287 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.102155 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:06.102623 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.102381 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-workload-socket" pod "5c1b682b-01eb-4717-9184-d81851ade4b0" (UID: "5c1b682b-01eb-4717-9184-d81851ade4b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:06.102786 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.102763 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-istio-data" pod "5c1b682b-01eb-4717-9184-d81851ade4b0" (UID: "5c1b682b-01eb-4717-9184-d81851ade4b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:06.104520 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.104458 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c1b682b-01eb-4717-9184-d81851ade4b0-istio-token" pod "5c1b682b-01eb-4717-9184-d81851ade4b0" (UID: "5c1b682b-01eb-4717-9184-d81851ade4b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:23:06.107065 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.107041 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c1b682b-01eb-4717-9184-d81851ade4b0-kube-api-access-nvctj" pod "5c1b682b-01eb-4717-9184-d81851ade4b0" (UID: "5c1b682b-01eb-4717-9184-d81851ade4b0"). InnerVolumeSpecName "kube-api-access-nvctj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:23:06.107431 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.107411 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-istio-envoy" pod "5c1b682b-01eb-4717-9184-d81851ade4b0" (UID: "5c1b682b-01eb-4717-9184-d81851ade4b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:06.107591 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.107558 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c1b682b-01eb-4717-9184-d81851ade4b0-istio-podinfo" pod "5c1b682b-01eb-4717-9184-d81851ade4b0" (UID: "5c1b682b-01eb-4717-9184-d81851ade4b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:23:06.203671 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.203636 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:06.203671 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.203667 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c1b682b-01eb-4717-9184-d81851ade4b0-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:06.203671 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.203675 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:06.203937 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.203686 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c1b682b-01eb-4717-9184-d81851ade4b0-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:06.203937 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.203695 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1b682b-01eb-4717-9184-d81851ade4b0-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:06.203937 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.203704 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvctj\" (UniqueName: \"kubernetes.io/projected/5c1b682b-01eb-4717-9184-d81851ade4b0-kube-api-access-nvctj\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:06.703558 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.698992 4951 generic.go:356] "Generic (PLEG): container finished" podID="5c1b682b-01eb-4717-9184-d81851ade4b0" containerID="3e52bfcd6536d75017df91674af8f7417a7a20b386b9711fe8361a4bd8676c15" exitCode=0 Oct 30 07:23:06.703558 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.699172 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" event={"ID":"5c1b682b-01eb-4717-9184-d81851ade4b0","Type":"ContainerDied","Data":"3e52bfcd6536d75017df91674af8f7417a7a20b386b9711fe8361a4bd8676c15"} Oct 30 07:23:06.703558 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.699224 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" event={"ID":"5c1b682b-01eb-4717-9184-d81851ade4b0","Type":"ContainerDied","Data":"7227635c5a8ae197117eb6c0367e827adfec48627a82872c2fcc252e0f7c3c29"} Oct 30 07:23:06.703558 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.699254 4951 scope.go:122] "RemoveContainer" containerID="3e52bfcd6536d75017df91674af8f7417a7a20b386b9711fe8361a4bd8676c15" Oct 30 07:23:06.703558 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.699455 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn" Oct 30 07:23:06.740029 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.739391 4951 scope.go:122] "RemoveContainer" containerID="fa2c7640c0bc198242c773e7da1f563fb2c1ccd7fbeec578cf012036992e4450" Oct 30 07:23:06.756790 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.756432 4951 scope.go:122] "RemoveContainer" containerID="3e52bfcd6536d75017df91674af8f7417a7a20b386b9711fe8361a4bd8676c15" Oct 30 07:23:06.757328 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:23:06.757039 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e52bfcd6536d75017df91674af8f7417a7a20b386b9711fe8361a4bd8676c15\": container with ID starting with 3e52bfcd6536d75017df91674af8f7417a7a20b386b9711fe8361a4bd8676c15 not found: ID does not exist" containerID="3e52bfcd6536d75017df91674af8f7417a7a20b386b9711fe8361a4bd8676c15" Oct 30 07:23:06.757328 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.757083 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e52bfcd6536d75017df91674af8f7417a7a20b386b9711fe8361a4bd8676c15"} err="failed to get container status \"3e52bfcd6536d75017df91674af8f7417a7a20b386b9711fe8361a4bd8676c15\": rpc error: code = NotFound desc = could not find container \"3e52bfcd6536d75017df91674af8f7417a7a20b386b9711fe8361a4bd8676c15\": container with ID starting with 3e52bfcd6536d75017df91674af8f7417a7a20b386b9711fe8361a4bd8676c15 not found: ID does not exist" Oct 30 07:23:06.757328 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.757107 4951 scope.go:122] "RemoveContainer" containerID="fa2c7640c0bc198242c773e7da1f563fb2c1ccd7fbeec578cf012036992e4450" Oct 30 07:23:06.758286 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:23:06.758038 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa2c7640c0bc198242c773e7da1f563fb2c1ccd7fbeec578cf012036992e4450\": container with ID starting with fa2c7640c0bc198242c773e7da1f563fb2c1ccd7fbeec578cf012036992e4450 not found: ID does not exist" containerID="fa2c7640c0bc198242c773e7da1f563fb2c1ccd7fbeec578cf012036992e4450" Oct 30 07:23:06.758286 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.758070 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2c7640c0bc198242c773e7da1f563fb2c1ccd7fbeec578cf012036992e4450"} err="failed to get container status \"fa2c7640c0bc198242c773e7da1f563fb2c1ccd7fbeec578cf012036992e4450\": rpc error: code = NotFound desc = could not find container \"fa2c7640c0bc198242c773e7da1f563fb2c1ccd7fbeec578cf012036992e4450\": container with ID starting with fa2c7640c0bc198242c773e7da1f563fb2c1ccd7fbeec578cf012036992e4450 not found: ID does not exist" Oct 30 07:23:06.767179 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.767140 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn"] Oct 30 07:23:06.778511 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:06.778476 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-efce7b-openshift-default-7858dc96f7-f2jwn"] Oct 30 07:23:07.774428 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:07.774389 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c1b682b-01eb-4717-9184-d81851ade4b0" path="/var/lib/kubelet/pods/5c1b682b-01eb-4717-9184-d81851ade4b0/volumes" Oct 30 07:23:07.974710 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:07.974676 4951 patch_prober.go:29] interesting pod/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:23:07.974898 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:07.974732 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" podUID="4f345a13-140a-4122-9ad1-ece45c401b58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:23:08.271196 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.271154 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1beb92-fc79c45ff-gpklb"] Oct 30 07:23:08.272143 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.272121 4951 cpu_manager.go:414] "containerMap: removing container" podUID="5c1b682b-01eb-4717-9184-d81851ade4b0" containerName="istio-proxy" Oct 30 07:23:08.272205 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.272147 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c1b682b-01eb-4717-9184-d81851ade4b0" containerName="istio-proxy" Oct 30 07:23:08.272205 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.272188 4951 cpu_manager.go:414] "containerMap: removing container" podUID="5c1b682b-01eb-4717-9184-d81851ade4b0" containerName="istio-proxy" Oct 30 07:23:08.272205 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.272196 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c1b682b-01eb-4717-9184-d81851ade4b0" containerName="istio-proxy" Oct 30 07:23:08.272399 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.272386 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c1b682b-01eb-4717-9184-d81851ade4b0" containerName="istio-proxy" Oct 30 07:23:08.272444 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.272407 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c1b682b-01eb-4717-9184-d81851ade4b0" containerName="istio-proxy" Oct 30 07:23:08.278520 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.278497 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1beb92-fc79c45ff-gpklb" Oct 30 07:23:08.297502 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.297467 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1beb92-fc79c45ff-gpklb"] Oct 30 07:23:08.304361 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.304328 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-1beb92\"" Oct 30 07:23:08.304510 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.304475 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-1beb92-dockercfg-s286r\"" Oct 30 07:23:08.327629 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.319853 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4"] Oct 30 07:23:08.353595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.351387 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/070d9aa8-7438-4866-83fc-ac9a7e3a0443-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-1beb92-fc79c45ff-gpklb\" (UID: \"070d9aa8-7438-4866-83fc-ac9a7e3a0443\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1beb92-fc79c45ff-gpklb" Oct 30 07:23:08.353595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.351463 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr9c6\" (UniqueName: \"kubernetes.io/projected/070d9aa8-7438-4866-83fc-ac9a7e3a0443-kube-api-access-xr9c6\") pod \"payload-pre-processing-e2e-praxis-a-1beb92-fc79c45ff-gpklb\" (UID: \"070d9aa8-7438-4866-83fc-ac9a7e3a0443\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1beb92-fc79c45ff-gpklb" Oct 30 07:23:08.363685 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.359550 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4" Oct 30 07:23:08.483721 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.452862 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b2cc236-34ca-4719-a938-2bf84d312d6e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4\" (UID: \"4b2cc236-34ca-4719-a938-2bf84d312d6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4" Oct 30 07:23:08.483721 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.452907 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwqjm\" (UniqueName: \"kubernetes.io/projected/4b2cc236-34ca-4719-a938-2bf84d312d6e-kube-api-access-nwqjm\") pod \"maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4\" (UID: \"4b2cc236-34ca-4719-a938-2bf84d312d6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4" Oct 30 07:23:08.483721 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.452998 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/070d9aa8-7438-4866-83fc-ac9a7e3a0443-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-1beb92-fc79c45ff-gpklb\" (UID: \"070d9aa8-7438-4866-83fc-ac9a7e3a0443\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1beb92-fc79c45ff-gpklb" Oct 30 07:23:08.483721 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.453042 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr9c6\" (UniqueName: \"kubernetes.io/projected/070d9aa8-7438-4866-83fc-ac9a7e3a0443-kube-api-access-xr9c6\") pod \"payload-pre-processing-e2e-praxis-a-1beb92-fc79c45ff-gpklb\" (UID: \"070d9aa8-7438-4866-83fc-ac9a7e3a0443\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1beb92-fc79c45ff-gpklb" Oct 30 07:23:08.483721 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.453072 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b2cc236-34ca-4719-a938-2bf84d312d6e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4\" (UID: \"4b2cc236-34ca-4719-a938-2bf84d312d6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4" Oct 30 07:23:08.483721 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.453208 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b2cc236-34ca-4719-a938-2bf84d312d6e-metrics-certs\") pod \"maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4\" (UID: \"4b2cc236-34ca-4719-a938-2bf84d312d6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4" Oct 30 07:23:08.483721 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.454214 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/070d9aa8-7438-4866-83fc-ac9a7e3a0443-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-1beb92-fc79c45ff-gpklb\" (UID: \"070d9aa8-7438-4866-83fc-ac9a7e3a0443\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1beb92-fc79c45ff-gpklb" Oct 30 07:23:08.549482 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.548545 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a8f428-serving-cert\"" Oct 30 07:23:08.559483 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.559444 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b2cc236-34ca-4719-a938-2bf84d312d6e-metrics-certs\") pod \"maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4\" (UID: \"4b2cc236-34ca-4719-a938-2bf84d312d6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4" Oct 30 07:23:08.559807 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.559528 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b2cc236-34ca-4719-a938-2bf84d312d6e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4\" (UID: \"4b2cc236-34ca-4719-a938-2bf84d312d6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4" Oct 30 07:23:08.559807 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.559556 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwqjm\" (UniqueName: \"kubernetes.io/projected/4b2cc236-34ca-4719-a938-2bf84d312d6e-kube-api-access-nwqjm\") pod \"maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4\" (UID: \"4b2cc236-34ca-4719-a938-2bf84d312d6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4" Oct 30 07:23:08.559807 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.559719 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b2cc236-34ca-4719-a938-2bf84d312d6e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4\" (UID: \"4b2cc236-34ca-4719-a938-2bf84d312d6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4" Oct 30 07:23:08.583771 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.582410 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b2cc236-34ca-4719-a938-2bf84d312d6e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4\" (UID: \"4b2cc236-34ca-4719-a938-2bf84d312d6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4" Oct 30 07:23:08.631357 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.591438 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b2cc236-34ca-4719-a938-2bf84d312d6e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4\" (UID: \"4b2cc236-34ca-4719-a938-2bf84d312d6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4" Oct 30 07:23:08.658856 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.658820 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b2cc236-34ca-4719-a938-2bf84d312d6e-metrics-certs\") pod \"maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4\" (UID: \"4b2cc236-34ca-4719-a938-2bf84d312d6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4" Oct 30 07:23:08.707312 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.706131 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwqjm\" (UniqueName: \"kubernetes.io/projected/4b2cc236-34ca-4719-a938-2bf84d312d6e-kube-api-access-nwqjm\") pod \"maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4\" (UID: \"4b2cc236-34ca-4719-a938-2bf84d312d6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4" Oct 30 07:23:08.723273 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.723236 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4"] Oct 30 07:23:08.731798 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.729595 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1beb92-8546db7586-cm9b4"] Oct 30 07:23:08.741635 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.736216 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr9c6\" (UniqueName: \"kubernetes.io/projected/070d9aa8-7438-4866-83fc-ac9a7e3a0443-kube-api-access-xr9c6\") pod \"payload-pre-processing-e2e-praxis-a-1beb92-fc79c45ff-gpklb\" (UID: \"070d9aa8-7438-4866-83fc-ac9a7e3a0443\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1beb92-fc79c45ff-gpklb" Oct 30 07:23:08.750845 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.749819 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1beb92-8546db7586-cm9b4"] Oct 30 07:23:08.750845 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.749942 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1beb92-8546db7586-cm9b4" Oct 30 07:23:08.767607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.767535 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4" Oct 30 07:23:08.864992 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.864889 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c506890-59f9-48b1-9d0c-c8878ffa730a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-1beb92-8546db7586-cm9b4\" (UID: \"2c506890-59f9-48b1-9d0c-c8878ffa730a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1beb92-8546db7586-cm9b4" Oct 30 07:23:08.865454 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.865168 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snqz6\" (UniqueName: \"kubernetes.io/projected/2c506890-59f9-48b1-9d0c-c8878ffa730a-kube-api-access-snqz6\") pod \"payload-processing-e2e-praxis-a-1beb92-8546db7586-cm9b4\" (UID: \"2c506890-59f9-48b1-9d0c-c8878ffa730a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1beb92-8546db7586-cm9b4" Oct 30 07:23:08.895831 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.895799 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1beb92-fc79c45ff-gpklb" Oct 30 07:23:08.970154 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.968785 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c506890-59f9-48b1-9d0c-c8878ffa730a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-1beb92-8546db7586-cm9b4\" (UID: \"2c506890-59f9-48b1-9d0c-c8878ffa730a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1beb92-8546db7586-cm9b4" Oct 30 07:23:08.970154 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.969670 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c506890-59f9-48b1-9d0c-c8878ffa730a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-1beb92-8546db7586-cm9b4\" (UID: \"2c506890-59f9-48b1-9d0c-c8878ffa730a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1beb92-8546db7586-cm9b4" Oct 30 07:23:08.970154 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:08.969870 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snqz6\" (UniqueName: \"kubernetes.io/projected/2c506890-59f9-48b1-9d0c-c8878ffa730a-kube-api-access-snqz6\") pod \"payload-processing-e2e-praxis-a-1beb92-8546db7586-cm9b4\" (UID: \"2c506890-59f9-48b1-9d0c-c8878ffa730a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1beb92-8546db7586-cm9b4" Oct 30 07:23:09.003236 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:09.003180 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snqz6\" (UniqueName: \"kubernetes.io/projected/2c506890-59f9-48b1-9d0c-c8878ffa730a-kube-api-access-snqz6\") pod \"payload-processing-e2e-praxis-a-1beb92-8546db7586-cm9b4\" (UID: \"2c506890-59f9-48b1-9d0c-c8878ffa730a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1beb92-8546db7586-cm9b4" Oct 30 07:23:09.071162 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:09.071125 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1beb92-8546db7586-cm9b4" Oct 30 07:23:09.095770 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:09.093013 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4"] Oct 30 07:23:09.104779 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:23:09.104746 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2cc236_34ca_4719_a938_2bf84d312d6e.slice/crio-bb350ef0ba11009c2242267816deb7be39a474fe8a9046d7ea40a5b298d1c301 WatchSource:0}: Error finding container bb350ef0ba11009c2242267816deb7be39a474fe8a9046d7ea40a5b298d1c301: Status 404 returned error can't find the container with id bb350ef0ba11009c2242267816deb7be39a474fe8a9046d7ea40a5b298d1c301 Oct 30 07:23:09.226941 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:09.226778 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1beb92-fc79c45ff-gpklb"] Oct 30 07:23:09.227559 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:23:09.227505 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod070d9aa8_7438_4866_83fc_ac9a7e3a0443.slice/crio-c5188740c528f6de0623c1c9bb097db822c10b6046f18fceca78ef83738b4c69 WatchSource:0}: Error finding container c5188740c528f6de0623c1c9bb097db822c10b6046f18fceca78ef83738b4c69: Status 404 returned error can't find the container with id c5188740c528f6de0623c1c9bb097db822c10b6046f18fceca78ef83738b4c69 Oct 30 07:23:09.404733 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:09.404695 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1beb92-8546db7586-cm9b4"] Oct 30 07:23:09.405968 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:23:09.405943 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c506890_59f9_48b1_9d0c_c8878ffa730a.slice/crio-8eeddb0c7bbedb5a5d593bab88d7b923dadec424170b69047b323d23267bd67e WatchSource:0}: Error finding container 8eeddb0c7bbedb5a5d593bab88d7b923dadec424170b69047b323d23267bd67e: Status 404 returned error can't find the container with id 8eeddb0c7bbedb5a5d593bab88d7b923dadec424170b69047b323d23267bd67e Oct 30 07:23:09.733148 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:09.733039 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4" event={"ID":"4b2cc236-34ca-4719-a938-2bf84d312d6e","Type":"ContainerStarted","Data":"b68e9367e957f21d68572c4f64f695fe0ce03480ea8247ed894464a60710b533"} Oct 30 07:23:09.733148 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:09.733120 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4" event={"ID":"4b2cc236-34ca-4719-a938-2bf84d312d6e","Type":"ContainerStarted","Data":"bb350ef0ba11009c2242267816deb7be39a474fe8a9046d7ea40a5b298d1c301"} Oct 30 07:23:09.733377 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:09.733151 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4" Oct 30 07:23:09.735151 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:09.735120 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1beb92-8546db7586-cm9b4" event={"ID":"2c506890-59f9-48b1-9d0c-c8878ffa730a","Type":"ContainerStarted","Data":"4aae1c9712fc15ad3f22d5ba6a59ff13304abeb036564e7f4384974a1b8f1d43"} Oct 30 07:23:09.735318 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:09.735176 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1beb92-8546db7586-cm9b4" event={"ID":"2c506890-59f9-48b1-9d0c-c8878ffa730a","Type":"ContainerStarted","Data":"8eeddb0c7bbedb5a5d593bab88d7b923dadec424170b69047b323d23267bd67e"} Oct 30 07:23:09.735318 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:09.735204 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-1beb92-8546db7586-cm9b4" Oct 30 07:23:09.737044 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:09.737010 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1beb92-fc79c45ff-gpklb" event={"ID":"070d9aa8-7438-4866-83fc-ac9a7e3a0443","Type":"ContainerStarted","Data":"19972948abc9b89d90c99a792ad81d9759c78e5fbfe994b4565a6e57a28c9e08"} Oct 30 07:23:09.737180 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:09.737063 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1beb92-fc79c45ff-gpklb" event={"ID":"070d9aa8-7438-4866-83fc-ac9a7e3a0443","Type":"ContainerStarted","Data":"c5188740c528f6de0623c1c9bb097db822c10b6046f18fceca78ef83738b4c69"} Oct 30 07:23:09.737180 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:09.737093 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1beb92-fc79c45ff-gpklb" Oct 30 07:23:09.778695 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:09.778635 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4" podStartSLOduration=1.4500352570000001 podStartE2EDuration="1.778616834s" totalImagesPullingTime="328.581577ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:23:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:23:09.775761766 +0000 UTC m=+1596.118809731" watchObservedRunningTime="2026-10-30 07:23:09.778616834 +0000 UTC m=+1596.121664800" Oct 30 07:23:09.816115 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:09.816070 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-1beb92-8546db7586-cm9b4" podStartSLOduration=1.816053597 podStartE2EDuration="1.816053597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:23:09.815703619 +0000 UTC m=+1596.158751595" watchObservedRunningTime="2026-10-30 07:23:09.816053597 +0000 UTC m=+1596.159101564" Oct 30 07:23:09.861764 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:09.861712 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1beb92-fc79c45ff-gpklb" podStartSLOduration=1.861698524 podStartE2EDuration="1.861698524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:23:09.855631431 +0000 UTC m=+1596.198679412" watchObservedRunningTime="2026-10-30 07:23:09.861698524 +0000 UTC m=+1596.204746514" Oct 30 07:23:10.756292 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:10.756251 4951 generic.go:356] "Generic (PLEG): container finished" podID="4f345a13-140a-4122-9ad1-ece45c401b58" containerID="285427ad353b770ac9cbaa25b34fcaae40926d5fa3e5bc9f62d634e61054a8bd" exitCode=0 Oct 30 07:23:10.756828 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:10.756317 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" event={"ID":"4f345a13-140a-4122-9ad1-ece45c401b58","Type":"ContainerDied","Data":"285427ad353b770ac9cbaa25b34fcaae40926d5fa3e5bc9f62d634e61054a8bd"} Oct 30 07:23:10.756883 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:10.756848 4951 scope.go:122] "RemoveContainer" containerID="285427ad353b770ac9cbaa25b34fcaae40926d5fa3e5bc9f62d634e61054a8bd" Oct 30 07:23:10.758522 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:10.758496 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:23:10.758622 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:10.758566 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:23:10.758672 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:10.758627 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:23:11.773711 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:11.773676 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" event={"ID":"4f345a13-140a-4122-9ad1-ece45c401b58","Type":"ContainerStarted","Data":"7b2ccf293ad0ef40186a3fe2ceba17aad67dd3b4aa992d9da406b748293160c7"} Oct 30 07:23:12.045030 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:12.044942 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:23:12.045413 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:12.045316 4951 patch_prober.go:29] interesting pod/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 07:23:12.045598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:12.045553 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" podUID="4f345a13-140a-4122-9ad1-ece45c401b58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 07:23:13.047050 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:13.047008 4951 patch_prober.go:29] interesting pod/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 07:23:13.047552 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:13.047083 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" podUID="4f345a13-140a-4122-9ad1-ece45c401b58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 07:23:13.662070 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:13.662033 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-597d57-6b875bfb88-trjgk"] Oct 30 07:23:13.699783 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:13.699746 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-7fb879d9f5-p2sqj"] Oct 30 07:23:14.054599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:14.050970 4951 patch_prober.go:29] interesting pod/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 07:23:14.054599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:14.051032 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" podUID="4f345a13-140a-4122-9ad1-ece45c401b58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 07:23:15.047070 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:15.047025 4951 patch_prober.go:29] interesting pod/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 07:23:15.047287 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:15.047094 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" podUID="4f345a13-140a-4122-9ad1-ece45c401b58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 07:23:15.763399 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:15.763364 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4" Oct 30 07:23:16.051664 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:16.050293 4951 patch_prober.go:29] interesting pod/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 07:23:16.051664 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:16.050352 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" podUID="4f345a13-140a-4122-9ad1-ece45c401b58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 07:23:17.043908 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:17.043868 4951 patch_prober.go:29] interesting pod/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 07:23:17.044331 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:17.043932 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" podUID="4f345a13-140a-4122-9ad1-ece45c401b58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 07:23:18.048438 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:18.048407 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:23:18.048922 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:18.048889 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:23:18.049787 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:18.049765 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:23:18.662684 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:18.662636 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-6b875bfb88-trjgk" podUID="274fd0e0-2519-4369-8e34-341772f6a37d" containerName="payload-processing" containerID="cri-o://d23edbaba1ad19377932578f87e4afc0b56a5a5e0f84c14198b96d7b7a70f2d7" gracePeriod=25 Oct 30 07:23:18.700562 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:18.700522 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-7fb879d9f5-p2sqj" podUID="6d253264-032c-4c43-92d6-340d3abbdeee" containerName="payload-pre-processing" containerID="cri-o://340986529dd79d69cdde366f11e4263b88992418d865d07e20a47d443d4d2e3e" gracePeriod=25 Oct 30 07:23:18.949467 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:18.949432 4951 generic.go:356] "Generic (PLEG): container finished" podID="6d253264-032c-4c43-92d6-340d3abbdeee" containerID="340986529dd79d69cdde366f11e4263b88992418d865d07e20a47d443d4d2e3e" exitCode=0 Oct 30 07:23:18.957561 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:18.952506 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-7fb879d9f5-p2sqj" event={"ID":"6d253264-032c-4c43-92d6-340d3abbdeee","Type":"ContainerDied","Data":"340986529dd79d69cdde366f11e4263b88992418d865d07e20a47d443d4d2e3e"} Oct 30 07:23:18.957561 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:18.956057 4951 generic.go:356] "Generic (PLEG): container finished" podID="274fd0e0-2519-4369-8e34-341772f6a37d" containerID="d23edbaba1ad19377932578f87e4afc0b56a5a5e0f84c14198b96d7b7a70f2d7" exitCode=0 Oct 30 07:23:18.957561 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:18.956843 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-6b875bfb88-trjgk" event={"ID":"274fd0e0-2519-4369-8e34-341772f6a37d","Type":"ContainerDied","Data":"d23edbaba1ad19377932578f87e4afc0b56a5a5e0f84c14198b96d7b7a70f2d7"} Oct 30 07:23:19.006392 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:19.006370 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-6b875bfb88-trjgk" Oct 30 07:23:19.046009 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:19.045905 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/274fd0e0-2519-4369-8e34-341772f6a37d-kube-api-access-bvvt2\" (UniqueName: \"kubernetes.io/projected/274fd0e0-2519-4369-8e34-341772f6a37d-kube-api-access-bvvt2\") pod \"274fd0e0-2519-4369-8e34-341772f6a37d\" (UID: \"274fd0e0-2519-4369-8e34-341772f6a37d\") " Oct 30 07:23:19.046141 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:19.046122 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/274fd0e0-2519-4369-8e34-341772f6a37d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/274fd0e0-2519-4369-8e34-341772f6a37d-plugins-config-volume\") pod \"274fd0e0-2519-4369-8e34-341772f6a37d\" (UID: \"274fd0e0-2519-4369-8e34-341772f6a37d\") " Oct 30 07:23:19.046750 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:19.046721 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/274fd0e0-2519-4369-8e34-341772f6a37d-plugins-config-volume" pod "274fd0e0-2519-4369-8e34-341772f6a37d" (UID: "274fd0e0-2519-4369-8e34-341772f6a37d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:23:19.048427 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:19.048405 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/274fd0e0-2519-4369-8e34-341772f6a37d-kube-api-access-bvvt2" pod "274fd0e0-2519-4369-8e34-341772f6a37d" (UID: "274fd0e0-2519-4369-8e34-341772f6a37d"). InnerVolumeSpecName "kube-api-access-bvvt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:23:19.072535 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:19.072507 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-7fb879d9f5-p2sqj" Oct 30 07:23:19.157917 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:19.157218 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d253264-032c-4c43-92d6-340d3abbdeee-kube-api-access-xjqb8\" (UniqueName: \"kubernetes.io/projected/6d253264-032c-4c43-92d6-340d3abbdeee-kube-api-access-xjqb8\") pod \"6d253264-032c-4c43-92d6-340d3abbdeee\" (UID: \"6d253264-032c-4c43-92d6-340d3abbdeee\") " Oct 30 07:23:19.158340 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:19.158289 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d253264-032c-4c43-92d6-340d3abbdeee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d253264-032c-4c43-92d6-340d3abbdeee-plugins-config-volume\") pod \"6d253264-032c-4c43-92d6-340d3abbdeee\" (UID: \"6d253264-032c-4c43-92d6-340d3abbdeee\") " Oct 30 07:23:19.159901 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:19.159862 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvvt2\" (UniqueName: \"kubernetes.io/projected/274fd0e0-2519-4369-8e34-341772f6a37d-kube-api-access-bvvt2\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:19.160081 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:19.160067 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/274fd0e0-2519-4369-8e34-341772f6a37d-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:19.160247 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:19.158904 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d253264-032c-4c43-92d6-340d3abbdeee-plugins-config-volume" pod "6d253264-032c-4c43-92d6-340d3abbdeee" (UID: "6d253264-032c-4c43-92d6-340d3abbdeee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:23:19.160783 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:19.160744 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d253264-032c-4c43-92d6-340d3abbdeee-kube-api-access-xjqb8" pod "6d253264-032c-4c43-92d6-340d3abbdeee" (UID: "6d253264-032c-4c43-92d6-340d3abbdeee"). InnerVolumeSpecName "kube-api-access-xjqb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:23:19.261637 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:19.261600 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjqb8\" (UniqueName: \"kubernetes.io/projected/6d253264-032c-4c43-92d6-340d3abbdeee-kube-api-access-xjqb8\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:19.261637 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:19.261637 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d253264-032c-4c43-92d6-340d3abbdeee-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:19.843714 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:23:19.843679 4951 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod274fd0e0_2519_4369_8e34_341772f6a37d.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d253264_032c_4c43_92d6_340d3abbdeee.slice/crio-7f0434f624f57767ca35cff40a25942c32213642f4e2771c34ba52a3f45db06b\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d253264_032c_4c43_92d6_340d3abbdeee.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod274fd0e0_2519_4369_8e34_341772f6a37d.slice/crio-fd91ef9c00c3147d710f6dc6bcf9137c05e0939aa8953a12cc05d9c3977a5929\": RecentStats: unable to find data in memory cache]" Oct 30 07:23:19.966937 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:19.966833 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-7fb879d9f5-p2sqj" Oct 30 07:23:19.966937 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:19.966856 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-7fb879d9f5-p2sqj" event={"ID":"6d253264-032c-4c43-92d6-340d3abbdeee","Type":"ContainerDied","Data":"7f0434f624f57767ca35cff40a25942c32213642f4e2771c34ba52a3f45db06b"} Oct 30 07:23:19.967188 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:19.966945 4951 scope.go:122] "RemoveContainer" containerID="340986529dd79d69cdde366f11e4263b88992418d865d07e20a47d443d4d2e3e" Oct 30 07:23:19.970748 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:19.970721 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-6b875bfb88-trjgk" Oct 30 07:23:19.972022 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:19.971791 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-6b875bfb88-trjgk" event={"ID":"274fd0e0-2519-4369-8e34-341772f6a37d","Type":"ContainerDied","Data":"fd91ef9c00c3147d710f6dc6bcf9137c05e0939aa8953a12cc05d9c3977a5929"} Oct 30 07:23:19.979464 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:19.979440 4951 scope.go:122] "RemoveContainer" containerID="d23edbaba1ad19377932578f87e4afc0b56a5a5e0f84c14198b96d7b7a70f2d7" Oct 30 07:23:20.019330 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:20.019297 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-597d57-6b875bfb88-trjgk"] Oct 30 07:23:20.032356 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:20.032320 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-597d57-6b875bfb88-trjgk"] Oct 30 07:23:20.058632 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:20.058587 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-7fb879d9f5-p2sqj"] Oct 30 07:23:20.064701 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:20.064671 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-7fb879d9f5-p2sqj"] Oct 30 07:23:20.759561 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:20.759526 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1beb92-fc79c45ff-gpklb" Oct 30 07:23:20.760116 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:20.759666 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-1beb92-8546db7586-cm9b4" Oct 30 07:23:21.775855 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:21.775821 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="274fd0e0-2519-4369-8e34-341772f6a37d" path="/var/lib/kubelet/pods/274fd0e0-2519-4369-8e34-341772f6a37d/volumes" Oct 30 07:23:21.776477 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:21.776456 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d253264-032c-4c43-92d6-340d3abbdeee" path="/var/lib/kubelet/pods/6d253264-032c-4c43-92d6-340d3abbdeee/volumes" Oct 30 07:23:23.023043 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:23:23.022999 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:23:28.659435 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:28.659394 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k"] Oct 30 07:23:28.659992 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:28.659664 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k" podUID="ce0e72c9-f992-4f9d-9da3-11ca496ac9e2" containerName="maas-api" containerID="cri-o://9194bcbd9b62bc5b9a3952318bbdff322aef4415c38f6ae461ee9f490f8c1e26" gracePeriod=30 Oct 30 07:23:28.818427 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:28.818393 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-579b5d699b-7g76s"] Oct 30 07:23:28.818663 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:28.818638 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-579b5d699b-7g76s" podUID="0b26b6b9-3f18-4090-b0e9-d2d537a03954" containerName="payload-processing" containerID="cri-o://4ea2f88384f4b461bbe2a0b1532a712a60fe8cf7c92b9c87f2d1de170b754e82" gracePeriod=30 Oct 30 07:23:29.063502 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.063463 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k" Oct 30 07:23:29.065959 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.065929 4951 generic.go:356] "Generic (PLEG): container finished" podID="ce0e72c9-f992-4f9d-9da3-11ca496ac9e2" containerID="9194bcbd9b62bc5b9a3952318bbdff322aef4415c38f6ae461ee9f490f8c1e26" exitCode=0 Oct 30 07:23:29.066099 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.065963 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k" Oct 30 07:23:29.066147 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.066122 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k" event={"ID":"ce0e72c9-f992-4f9d-9da3-11ca496ac9e2","Type":"ContainerDied","Data":"9194bcbd9b62bc5b9a3952318bbdff322aef4415c38f6ae461ee9f490f8c1e26"} Oct 30 07:23:29.066185 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.066164 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k" event={"ID":"ce0e72c9-f992-4f9d-9da3-11ca496ac9e2","Type":"ContainerDied","Data":"4e7158ea1e42fd8ebc5f413044e54101b27fd738f48692b7649c1cddfc63a55b"} Oct 30 07:23:29.066219 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.066192 4951 scope.go:122] "RemoveContainer" containerID="9194bcbd9b62bc5b9a3952318bbdff322aef4415c38f6ae461ee9f490f8c1e26" Oct 30 07:23:29.068161 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.068134 4951 generic.go:356] "Generic (PLEG): container finished" podID="0b26b6b9-3f18-4090-b0e9-d2d537a03954" containerID="4ea2f88384f4b461bbe2a0b1532a712a60fe8cf7c92b9c87f2d1de170b754e82" exitCode=0 Oct 30 07:23:29.068287 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.068196 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-579b5d699b-7g76s" event={"ID":"0b26b6b9-3f18-4090-b0e9-d2d537a03954","Type":"ContainerDied","Data":"4ea2f88384f4b461bbe2a0b1532a712a60fe8cf7c92b9c87f2d1de170b754e82"} Oct 30 07:23:29.090088 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.089793 4951 scope.go:122] "RemoveContainer" containerID="9194bcbd9b62bc5b9a3952318bbdff322aef4415c38f6ae461ee9f490f8c1e26" Oct 30 07:23:29.090190 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:23:29.090159 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9194bcbd9b62bc5b9a3952318bbdff322aef4415c38f6ae461ee9f490f8c1e26\": container with ID starting with 9194bcbd9b62bc5b9a3952318bbdff322aef4415c38f6ae461ee9f490f8c1e26 not found: ID does not exist" containerID="9194bcbd9b62bc5b9a3952318bbdff322aef4415c38f6ae461ee9f490f8c1e26" Oct 30 07:23:29.090265 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.090207 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9194bcbd9b62bc5b9a3952318bbdff322aef4415c38f6ae461ee9f490f8c1e26"} err="failed to get container status \"9194bcbd9b62bc5b9a3952318bbdff322aef4415c38f6ae461ee9f490f8c1e26\": rpc error: code = NotFound desc = could not find container \"9194bcbd9b62bc5b9a3952318bbdff322aef4415c38f6ae461ee9f490f8c1e26\": container with ID starting with 9194bcbd9b62bc5b9a3952318bbdff322aef4415c38f6ae461ee9f490f8c1e26 not found: ID does not exist" Oct 30 07:23:29.193607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.192904 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce0e72c9-f992-4f9d-9da3-11ca496ac9e2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0e72c9-f992-4f9d-9da3-11ca496ac9e2-maas-api-tls\") pod \"ce0e72c9-f992-4f9d-9da3-11ca496ac9e2\" (UID: \"ce0e72c9-f992-4f9d-9da3-11ca496ac9e2\") " Oct 30 07:23:29.193607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.192961 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce0e72c9-f992-4f9d-9da3-11ca496ac9e2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0e72c9-f992-4f9d-9da3-11ca496ac9e2-openshift-service-ca\") pod \"ce0e72c9-f992-4f9d-9da3-11ca496ac9e2\" (UID: \"ce0e72c9-f992-4f9d-9da3-11ca496ac9e2\") " Oct 30 07:23:29.193607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.193196 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce0e72c9-f992-4f9d-9da3-11ca496ac9e2-kube-api-access-gps9f\" (UniqueName: \"kubernetes.io/projected/ce0e72c9-f992-4f9d-9da3-11ca496ac9e2-kube-api-access-gps9f\") pod \"ce0e72c9-f992-4f9d-9da3-11ca496ac9e2\" (UID: \"ce0e72c9-f992-4f9d-9da3-11ca496ac9e2\") " Oct 30 07:23:29.193607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.193234 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce0e72c9-f992-4f9d-9da3-11ca496ac9e2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0e72c9-f992-4f9d-9da3-11ca496ac9e2-metrics-certs\") pod \"ce0e72c9-f992-4f9d-9da3-11ca496ac9e2\" (UID: \"ce0e72c9-f992-4f9d-9da3-11ca496ac9e2\") " Oct 30 07:23:29.240660 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.230204 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce0e72c9-f992-4f9d-9da3-11ca496ac9e2-openshift-service-ca" pod "ce0e72c9-f992-4f9d-9da3-11ca496ac9e2" (UID: "ce0e72c9-f992-4f9d-9da3-11ca496ac9e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:23:29.265607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.258235 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce0e72c9-f992-4f9d-9da3-11ca496ac9e2-kube-api-access-gps9f" pod "ce0e72c9-f992-4f9d-9da3-11ca496ac9e2" (UID: "ce0e72c9-f992-4f9d-9da3-11ca496ac9e2"). InnerVolumeSpecName "kube-api-access-gps9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:23:29.281597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.266500 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce0e72c9-f992-4f9d-9da3-11ca496ac9e2-metrics-certs" pod "ce0e72c9-f992-4f9d-9da3-11ca496ac9e2" (UID: "ce0e72c9-f992-4f9d-9da3-11ca496ac9e2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:23:29.302593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.296661 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0e72c9-f992-4f9d-9da3-11ca496ac9e2-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:29.302593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.296696 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gps9f\" (UniqueName: \"kubernetes.io/projected/ce0e72c9-f992-4f9d-9da3-11ca496ac9e2-kube-api-access-gps9f\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:29.302593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.296711 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0e72c9-f992-4f9d-9da3-11ca496ac9e2-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:29.334601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.332061 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw"] Oct 30 07:23:29.459610 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.453264 4951 cpu_manager.go:414] "containerMap: removing container" podUID="6d253264-032c-4c43-92d6-340d3abbdeee" containerName="payload-pre-processing" Oct 30 07:23:29.459610 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.453311 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d253264-032c-4c43-92d6-340d3abbdeee" containerName="payload-pre-processing" Oct 30 07:23:29.459610 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.453418 4951 cpu_manager.go:414] "containerMap: removing container" podUID="274fd0e0-2519-4369-8e34-341772f6a37d" containerName="payload-processing" Oct 30 07:23:29.459610 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.453430 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="274fd0e0-2519-4369-8e34-341772f6a37d" containerName="payload-processing" Oct 30 07:23:29.459610 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.453487 4951 cpu_manager.go:414] "containerMap: removing container" podUID="ce0e72c9-f992-4f9d-9da3-11ca496ac9e2" containerName="maas-api" Oct 30 07:23:29.459610 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.453496 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce0e72c9-f992-4f9d-9da3-11ca496ac9e2" containerName="maas-api" Oct 30 07:23:29.459610 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.457492 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="274fd0e0-2519-4369-8e34-341772f6a37d" containerName="payload-processing" Oct 30 07:23:29.459610 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.457545 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce0e72c9-f992-4f9d-9da3-11ca496ac9e2" containerName="maas-api" Oct 30 07:23:29.459610 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.457813 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d253264-032c-4c43-92d6-340d3abbdeee" containerName="payload-pre-processing" Oct 30 07:23:29.463726 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.462954 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce0e72c9-f992-4f9d-9da3-11ca496ac9e2-maas-api-tls" pod "ce0e72c9-f992-4f9d-9da3-11ca496ac9e2" (UID: "ce0e72c9-f992-4f9d-9da3-11ca496ac9e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:23:29.470712 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.470686 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw"] Oct 30 07:23:29.470850 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.470774 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw" Oct 30 07:23:29.480455 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.480427 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-579b5d699b-7g76s" Oct 30 07:23:29.485499 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.485478 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-1beb92-serving-cert\"" Oct 30 07:23:29.581594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.581083 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b26b6b9-3f18-4090-b0e9-d2d537a03954-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b26b6b9-3f18-4090-b0e9-d2d537a03954-plugins-config-volume\") pod \"0b26b6b9-3f18-4090-b0e9-d2d537a03954\" (UID: \"0b26b6b9-3f18-4090-b0e9-d2d537a03954\") " Oct 30 07:23:29.581594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.581269 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b26b6b9-3f18-4090-b0e9-d2d537a03954-kube-api-access-4frsv\" (UniqueName: \"kubernetes.io/projected/0b26b6b9-3f18-4090-b0e9-d2d537a03954-kube-api-access-4frsv\") pod \"0b26b6b9-3f18-4090-b0e9-d2d537a03954\" (UID: \"0b26b6b9-3f18-4090-b0e9-d2d537a03954\") " Oct 30 07:23:29.581851 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.581646 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e312e239-4207-479a-a33e-d7769edd58b5-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw\" (UID: \"e312e239-4207-479a-a33e-d7769edd58b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw" Oct 30 07:23:29.581851 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.581695 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e312e239-4207-479a-a33e-d7769edd58b5-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw\" (UID: \"e312e239-4207-479a-a33e-d7769edd58b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw" Oct 30 07:23:29.581851 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.581778 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e312e239-4207-479a-a33e-d7769edd58b5-metrics-certs\") pod \"maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw\" (UID: \"e312e239-4207-479a-a33e-d7769edd58b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw" Oct 30 07:23:29.582022 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.581928 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb5tk\" (UniqueName: \"kubernetes.io/projected/e312e239-4207-479a-a33e-d7769edd58b5-kube-api-access-gb5tk\") pod \"maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw\" (UID: \"e312e239-4207-479a-a33e-d7769edd58b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw" Oct 30 07:23:29.582446 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.582223 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0e72c9-f992-4f9d-9da3-11ca496ac9e2-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:29.582446 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.582414 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b26b6b9-3f18-4090-b0e9-d2d537a03954-plugins-config-volume" pod "0b26b6b9-3f18-4090-b0e9-d2d537a03954" (UID: "0b26b6b9-3f18-4090-b0e9-d2d537a03954"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:23:29.602319 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.600799 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b26b6b9-3f18-4090-b0e9-d2d537a03954-kube-api-access-4frsv" pod "0b26b6b9-3f18-4090-b0e9-d2d537a03954" (UID: "0b26b6b9-3f18-4090-b0e9-d2d537a03954"). InnerVolumeSpecName "kube-api-access-4frsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:23:29.683008 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.682974 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e312e239-4207-479a-a33e-d7769edd58b5-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw\" (UID: \"e312e239-4207-479a-a33e-d7769edd58b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw" Oct 30 07:23:29.683579 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.683037 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e312e239-4207-479a-a33e-d7769edd58b5-metrics-certs\") pod \"maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw\" (UID: \"e312e239-4207-479a-a33e-d7769edd58b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw" Oct 30 07:23:29.683579 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.683123 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb5tk\" (UniqueName: \"kubernetes.io/projected/e312e239-4207-479a-a33e-d7769edd58b5-kube-api-access-gb5tk\") pod \"maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw\" (UID: \"e312e239-4207-479a-a33e-d7769edd58b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw" Oct 30 07:23:29.683579 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.683221 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e312e239-4207-479a-a33e-d7769edd58b5-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw\" (UID: \"e312e239-4207-479a-a33e-d7769edd58b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw" Oct 30 07:23:29.683579 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.683270 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4frsv\" (UniqueName: \"kubernetes.io/projected/0b26b6b9-3f18-4090-b0e9-d2d537a03954-kube-api-access-4frsv\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:29.683579 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.683281 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b26b6b9-3f18-4090-b0e9-d2d537a03954-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:29.684271 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.684250 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e312e239-4207-479a-a33e-d7769edd58b5-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw\" (UID: \"e312e239-4207-479a-a33e-d7769edd58b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw" Oct 30 07:23:29.686392 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.686366 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e312e239-4207-479a-a33e-d7769edd58b5-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw\" (UID: \"e312e239-4207-479a-a33e-d7769edd58b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw" Oct 30 07:23:29.686884 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.686860 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e312e239-4207-479a-a33e-d7769edd58b5-metrics-certs\") pod \"maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw\" (UID: \"e312e239-4207-479a-a33e-d7769edd58b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw" Oct 30 07:23:29.707827 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.707800 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb5tk\" (UniqueName: \"kubernetes.io/projected/e312e239-4207-479a-a33e-d7769edd58b5-kube-api-access-gb5tk\") pod \"maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw\" (UID: \"e312e239-4207-479a-a33e-d7769edd58b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw" Oct 30 07:23:29.774702 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.774671 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k"] Oct 30 07:23:29.774702 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.774706 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k"] Oct 30 07:23:29.791763 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:29.791741 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw" Oct 30 07:23:30.086379 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:30.086336 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-579b5d699b-7g76s" Oct 30 07:23:30.086729 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:30.086697 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-579b5d699b-7g76s" event={"ID":"0b26b6b9-3f18-4090-b0e9-d2d537a03954","Type":"ContainerDied","Data":"1b80f6924bf55e531f38a4de2ce7922e8590e7abf0e60ebca61500c58851f523"} Oct 30 07:23:30.087270 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:30.087248 4951 scope.go:122] "RemoveContainer" containerID="4ea2f88384f4b461bbe2a0b1532a712a60fe8cf7c92b9c87f2d1de170b754e82" Oct 30 07:23:30.184801 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:30.184769 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw"] Oct 30 07:23:30.194267 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:30.194176 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-579b5d699b-7g76s"] Oct 30 07:23:30.203005 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:30.202977 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-579b5d699b-7g76s"] Oct 30 07:23:31.108985 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:31.108941 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw" event={"ID":"e312e239-4207-479a-a33e-d7769edd58b5","Type":"ContainerStarted","Data":"654ebbd2ccf089d6010bd315dc78fd512df44f970b55254ac4a87e16bfb51e6d"} Oct 30 07:23:31.109501 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:31.109022 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw" Oct 30 07:23:31.109501 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:31.109057 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw" event={"ID":"e312e239-4207-479a-a33e-d7769edd58b5","Type":"ContainerStarted","Data":"53066f4a077c0f714183c01ff3caf21d013a6b5f40356487e5bd7f15e53472b5"} Oct 30 07:23:31.776310 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:31.776272 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b26b6b9-3f18-4090-b0e9-d2d537a03954" path="/var/lib/kubelet/pods/0b26b6b9-3f18-4090-b0e9-d2d537a03954/volumes" Oct 30 07:23:31.776914 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:31.776889 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce0e72c9-f992-4f9d-9da3-11ca496ac9e2" path="/var/lib/kubelet/pods/ce0e72c9-f992-4f9d-9da3-11ca496ac9e2/volumes" Oct 30 07:23:31.876670 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:31.876628 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ab3d83-ffdd7c8d9-ghs7k" podUID="ce0e72c9-f992-4f9d-9da3-11ca496ac9e2" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.127:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:23:34.190333 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:34.190263 4951 generic.go:356] "Generic (PLEG): container finished" podID="03884451-d62c-4eea-827f-6e0c9359cbec" containerID="9c409058287aed26d6bcb72f9661a2627c4114bc51e907727cba31aca4eab50c" exitCode=0 Oct 30 07:23:34.190745 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:34.190337 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" event={"ID":"03884451-d62c-4eea-827f-6e0c9359cbec","Type":"ContainerDied","Data":"9c409058287aed26d6bcb72f9661a2627c4114bc51e907727cba31aca4eab50c"} Oct 30 07:23:34.190786 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:34.190764 4951 scope.go:122] "RemoveContainer" containerID="9c409058287aed26d6bcb72f9661a2627c4114bc51e907727cba31aca4eab50c" Oct 30 07:23:34.192225 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:34.192195 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:23:34.192290 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:34.192264 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:23:34.192327 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:34.192299 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:23:34.261131 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:34.261080 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw" podStartSLOduration=4.971858807 podStartE2EDuration="5.261062973s" totalImagesPullingTime="289.204166ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:23:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:23:31.151464113 +0000 UTC m=+1617.494512078" watchObservedRunningTime="2026-10-30 07:23:34.261062973 +0000 UTC m=+1620.604110976" Oct 30 07:23:35.199506 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:35.199415 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" event={"ID":"03884451-d62c-4eea-827f-6e0c9359cbec","Type":"ContainerStarted","Data":"f860e49acd45a39557899aea264c0ffebaeb49db89cca82926091656add070f1"} Oct 30 07:23:35.249726 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:35.249697 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:23:35.250004 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:35.249979 4951 patch_prober.go:29] interesting pod/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 07:23:35.250069 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:35.250047 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" podUID="03884451-d62c-4eea-827f-6e0c9359cbec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 07:23:36.266617 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:36.263046 4951 patch_prober.go:29] interesting pod/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 07:23:36.266617 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:36.263113 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" podUID="03884451-d62c-4eea-827f-6e0c9359cbec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 07:23:36.583824 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:36.583737 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-866967ccd7-hnk4l"] Oct 30 07:23:36.584477 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:36.584455 4951 cpu_manager.go:414] "containerMap: removing container" podUID="0b26b6b9-3f18-4090-b0e9-d2d537a03954" containerName="payload-processing" Oct 30 07:23:36.584633 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:36.584479 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b26b6b9-3f18-4090-b0e9-d2d537a03954" containerName="payload-processing" Oct 30 07:23:36.584735 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:36.584718 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b26b6b9-3f18-4090-b0e9-d2d537a03954" containerName="payload-processing" Oct 30 07:23:36.588488 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:36.588468 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-866967ccd7-hnk4l" Oct 30 07:23:36.615456 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:36.615412 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-597d57-dockercfg-db5dg\"" Oct 30 07:23:36.615691 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:36.615476 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-597d57\"" Oct 30 07:23:36.620005 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:36.619971 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-866967ccd7-hnk4l"] Oct 30 07:23:36.684595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:36.682498 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e0fd956-de2f-4c18-b132-272da9769644-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-597d57-866967ccd7-hnk4l\" (UID: \"8e0fd956-de2f-4c18-b132-272da9769644\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-866967ccd7-hnk4l" Oct 30 07:23:36.684595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:36.682765 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww54v\" (UniqueName: \"kubernetes.io/projected/8e0fd956-de2f-4c18-b132-272da9769644-kube-api-access-ww54v\") pod \"payload-pre-processing-e2e-ipp-b-597d57-866967ccd7-hnk4l\" (UID: \"8e0fd956-de2f-4c18-b132-272da9769644\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-866967ccd7-hnk4l" Oct 30 07:23:36.818591 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:36.818480 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e0fd956-de2f-4c18-b132-272da9769644-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-597d57-866967ccd7-hnk4l\" (UID: \"8e0fd956-de2f-4c18-b132-272da9769644\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-866967ccd7-hnk4l" Oct 30 07:23:36.835731 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:36.835645 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e0fd956-de2f-4c18-b132-272da9769644-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-597d57-866967ccd7-hnk4l\" (UID: \"8e0fd956-de2f-4c18-b132-272da9769644\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-866967ccd7-hnk4l" Oct 30 07:23:36.836252 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:36.836217 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww54v\" (UniqueName: \"kubernetes.io/projected/8e0fd956-de2f-4c18-b132-272da9769644-kube-api-access-ww54v\") pod \"payload-pre-processing-e2e-ipp-b-597d57-866967ccd7-hnk4l\" (UID: \"8e0fd956-de2f-4c18-b132-272da9769644\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-866967ccd7-hnk4l" Oct 30 07:23:36.949848 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:36.906784 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww54v\" (UniqueName: \"kubernetes.io/projected/8e0fd956-de2f-4c18-b132-272da9769644-kube-api-access-ww54v\") pod \"payload-pre-processing-e2e-ipp-b-597d57-866967ccd7-hnk4l\" (UID: \"8e0fd956-de2f-4c18-b132-272da9769644\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-866967ccd7-hnk4l" Oct 30 07:23:37.013597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:37.010632 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-597d57-7889966d58-vb8fv"] Oct 30 07:23:37.153651 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:37.151992 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-7889966d58-vb8fv" Oct 30 07:23:37.160805 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:37.160767 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw" Oct 30 07:23:37.185348 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:37.170968 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-597d57-7889966d58-vb8fv"] Oct 30 07:23:37.255153 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:37.249731 4951 patch_prober.go:29] interesting pod/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 07:23:37.255153 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:37.249848 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" podUID="03884451-d62c-4eea-827f-6e0c9359cbec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 07:23:37.326351 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:37.326098 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-866967ccd7-hnk4l" Oct 30 07:23:37.327379 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:37.326966 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4152dc51-db44-4f78-b551-3310cf6137c2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-597d57-7889966d58-vb8fv\" (UID: \"4152dc51-db44-4f78-b551-3310cf6137c2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-7889966d58-vb8fv" Oct 30 07:23:37.327379 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:37.327079 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57lzq\" (UniqueName: \"kubernetes.io/projected/4152dc51-db44-4f78-b551-3310cf6137c2-kube-api-access-57lzq\") pod \"payload-processing-e2e-ipp-b-597d57-7889966d58-vb8fv\" (UID: \"4152dc51-db44-4f78-b551-3310cf6137c2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-7889966d58-vb8fv" Oct 30 07:23:37.442563 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:37.431971 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4152dc51-db44-4f78-b551-3310cf6137c2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-597d57-7889966d58-vb8fv\" (UID: \"4152dc51-db44-4f78-b551-3310cf6137c2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-7889966d58-vb8fv" Oct 30 07:23:37.442563 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:37.432083 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57lzq\" (UniqueName: \"kubernetes.io/projected/4152dc51-db44-4f78-b551-3310cf6137c2-kube-api-access-57lzq\") pod \"payload-processing-e2e-ipp-b-597d57-7889966d58-vb8fv\" (UID: \"4152dc51-db44-4f78-b551-3310cf6137c2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-7889966d58-vb8fv" Oct 30 07:23:37.442563 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:37.434314 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4152dc51-db44-4f78-b551-3310cf6137c2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-597d57-7889966d58-vb8fv\" (UID: \"4152dc51-db44-4f78-b551-3310cf6137c2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-7889966d58-vb8fv" Oct 30 07:23:37.444604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:37.444540 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx"] Oct 30 07:23:37.445067 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:37.445011 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" podUID="0c51720b-1e0b-4cb0-a1e2-08065c91acde" containerName="istio-proxy" containerID="cri-o://1cd7098c17f3398406adb9999819c2ead1525e521622cc70fa526329f51984e4" gracePeriod=30 Oct 30 07:23:37.491419 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:37.491389 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57lzq\" (UniqueName: \"kubernetes.io/projected/4152dc51-db44-4f78-b551-3310cf6137c2-kube-api-access-57lzq\") pod \"payload-processing-e2e-ipp-b-597d57-7889966d58-vb8fv\" (UID: \"4152dc51-db44-4f78-b551-3310cf6137c2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-7889966d58-vb8fv" Oct 30 07:23:37.624104 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:37.624046 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-7889966d58-vb8fv" Oct 30 07:23:38.163231 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:38.162935 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-866967ccd7-hnk4l"] Oct 30 07:23:38.206735 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:38.206705 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-597d57-7889966d58-vb8fv"] Oct 30 07:23:38.208136 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:23:38.208103 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4152dc51_db44_4f78_b551_3310cf6137c2.slice/crio-65b38b921d8c816063bc36d2873f6efb030d1545a226bd501d0acf4546323177 WatchSource:0}: Error finding container 65b38b921d8c816063bc36d2873f6efb030d1545a226bd501d0acf4546323177: Status 404 returned error can't find the container with id 65b38b921d8c816063bc36d2873f6efb030d1545a226bd501d0acf4546323177 Oct 30 07:23:38.261257 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:38.261214 4951 patch_prober.go:29] interesting pod/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 07:23:38.261386 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:38.261279 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" podUID="03884451-d62c-4eea-827f-6e0c9359cbec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 07:23:38.371654 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:38.367048 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-866967ccd7-hnk4l" event={"ID":"8e0fd956-de2f-4c18-b132-272da9769644","Type":"ContainerStarted","Data":"c6007e510466393655518f1e7915098c4582219714232af61b783dcc09a11cdf"} Oct 30 07:23:38.371654 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:38.367131 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-866967ccd7-hnk4l" event={"ID":"8e0fd956-de2f-4c18-b132-272da9769644","Type":"ContainerStarted","Data":"70617db797a9bdea3594aac15e66cbd35caffcb62ab1f2dfb4f44de2c95c1956"} Oct 30 07:23:38.371654 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:38.367181 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-866967ccd7-hnk4l" Oct 30 07:23:38.371654 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:38.368752 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-7889966d58-vb8fv" event={"ID":"4152dc51-db44-4f78-b551-3310cf6137c2","Type":"ContainerStarted","Data":"65b38b921d8c816063bc36d2873f6efb030d1545a226bd501d0acf4546323177"} Oct 30 07:23:39.259596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:39.253961 4951 patch_prober.go:29] interesting pod/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 07:23:39.259596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:39.254027 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" podUID="03884451-d62c-4eea-827f-6e0c9359cbec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 07:23:39.385616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:39.385566 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-7889966d58-vb8fv" event={"ID":"4152dc51-db44-4f78-b551-3310cf6137c2","Type":"ContainerStarted","Data":"b64094dd5b601f963d574d8f4c0adda84cbc1ef565c298b75731a28c7a8b7e24"} Oct 30 07:23:39.386254 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:39.386234 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-7889966d58-vb8fv" Oct 30 07:23:39.433598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:39.432989 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-7889966d58-vb8fv" podStartSLOduration=3.432965302 podStartE2EDuration="3.432965302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:23:39.431131454 +0000 UTC m=+1625.774179420" watchObservedRunningTime="2026-10-30 07:23:39.432965302 +0000 UTC m=+1625.776013272" Oct 30 07:23:39.459597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:39.455803 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-866967ccd7-hnk4l" podStartSLOduration=3.455768989 podStartE2EDuration="3.455768989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:23:38.411683952 +0000 UTC m=+1624.754731918" watchObservedRunningTime="2026-10-30 07:23:39.455768989 +0000 UTC m=+1625.798816957" Oct 30 07:23:40.252597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:40.251513 4951 patch_prober.go:29] interesting pod/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 07:23:40.252597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:40.251589 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" podUID="03884451-d62c-4eea-827f-6e0c9359cbec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 07:23:41.249595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:41.249036 4951 patch_prober.go:29] interesting pod/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 07:23:41.249595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:41.249115 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" podUID="03884451-d62c-4eea-827f-6e0c9359cbec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 07:23:42.499589 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:42.499539 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:23:42.500058 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:42.500036 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:23:42.501062 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:42.501038 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:23:42.849802 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:42.849761 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:23:42.870308 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:42.870271 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istio-data\") pod \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " Oct 30 07:23:42.870476 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:42.870322 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istio-podinfo\") pod \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " Oct 30 07:23:42.870476 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:42.870384 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-credential-socket\") pod \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " Oct 30 07:23:42.870476 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:42.870417 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c51720b-1e0b-4cb0-a1e2-08065c91acde-kube-api-access-8zfhw\" (UniqueName: \"kubernetes.io/projected/0c51720b-1e0b-4cb0-a1e2-08065c91acde-kube-api-access-8zfhw\") pod \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " Oct 30 07:23:42.870476 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:42.870445 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istio-envoy\") pod \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " Oct 30 07:23:42.870630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:42.870498 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istiod-ca-cert\") pod \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " Oct 30 07:23:42.870630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:42.870567 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istio-token\" (UniqueName: \"kubernetes.io/projected/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istio-token\") pod \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " Oct 30 07:23:42.870709 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:42.870628 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-workload-socket\") pod \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " Oct 30 07:23:42.870709 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:42.870680 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-workload-certs\") pod \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\" (UID: \"0c51720b-1e0b-4cb0-a1e2-08065c91acde\") " Oct 30 07:23:42.871238 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:42.871219 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-workload-certs" pod "0c51720b-1e0b-4cb0-a1e2-08065c91acde" (UID: "0c51720b-1e0b-4cb0-a1e2-08065c91acde"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:42.871422 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:42.871407 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istio-data" pod "0c51720b-1e0b-4cb0-a1e2-08065c91acde" (UID: "0c51720b-1e0b-4cb0-a1e2-08065c91acde"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:42.871838 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:42.871808 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-credential-socket" pod "0c51720b-1e0b-4cb0-a1e2-08065c91acde" (UID: "0c51720b-1e0b-4cb0-a1e2-08065c91acde"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:42.872002 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:42.871836 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istiod-ca-cert" pod "0c51720b-1e0b-4cb0-a1e2-08065c91acde" (UID: "0c51720b-1e0b-4cb0-a1e2-08065c91acde"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:23:42.872076 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:42.872022 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-workload-socket" pod "0c51720b-1e0b-4cb0-a1e2-08065c91acde" (UID: "0c51720b-1e0b-4cb0-a1e2-08065c91acde"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:42.873953 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:42.873922 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istio-token" pod "0c51720b-1e0b-4cb0-a1e2-08065c91acde" (UID: "0c51720b-1e0b-4cb0-a1e2-08065c91acde"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:23:42.874080 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:42.873958 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c51720b-1e0b-4cb0-a1e2-08065c91acde-kube-api-access-8zfhw" pod "0c51720b-1e0b-4cb0-a1e2-08065c91acde" (UID: "0c51720b-1e0b-4cb0-a1e2-08065c91acde"). InnerVolumeSpecName "kube-api-access-8zfhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:23:42.874247 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:42.874227 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istio-podinfo" pod "0c51720b-1e0b-4cb0-a1e2-08065c91acde" (UID: "0c51720b-1e0b-4cb0-a1e2-08065c91acde"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:23:42.874323 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:42.874258 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istio-envoy" pod "0c51720b-1e0b-4cb0-a1e2-08065c91acde" (UID: "0c51720b-1e0b-4cb0-a1e2-08065c91acde"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:42.972755 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:42.972600 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zfhw\" (UniqueName: \"kubernetes.io/projected/0c51720b-1e0b-4cb0-a1e2-08065c91acde-kube-api-access-8zfhw\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:42.972755 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:42.972635 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:42.972755 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:42.972650 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:42.972755 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:42.972665 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:42.972755 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:42.972683 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:42.972755 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:42.972698 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:42.972755 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:42.972710 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:42.972755 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:42.972723 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c51720b-1e0b-4cb0-a1e2-08065c91acde-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:42.972755 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:42.972736 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c51720b-1e0b-4cb0-a1e2-08065c91acde-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:43.449599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:43.432694 4951 generic.go:356] "Generic (PLEG): container finished" podID="0c51720b-1e0b-4cb0-a1e2-08065c91acde" containerID="1cd7098c17f3398406adb9999819c2ead1525e521622cc70fa526329f51984e4" exitCode=0 Oct 30 07:23:43.449599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:43.432919 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" Oct 30 07:23:43.449599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:43.433069 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" event={"ID":"0c51720b-1e0b-4cb0-a1e2-08065c91acde","Type":"ContainerDied","Data":"1cd7098c17f3398406adb9999819c2ead1525e521622cc70fa526329f51984e4"} Oct 30 07:23:43.449599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:43.433124 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx" event={"ID":"0c51720b-1e0b-4cb0-a1e2-08065c91acde","Type":"ContainerDied","Data":"ec546b376dcac1a961fa6fe4d1be3bda5229706e19df080abeb67cd9eccccc16"} Oct 30 07:23:43.449599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:43.433156 4951 scope.go:122] "RemoveContainer" containerID="1cd7098c17f3398406adb9999819c2ead1525e521622cc70fa526329f51984e4" Oct 30 07:23:43.468120 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:43.468094 4951 scope.go:122] "RemoveContainer" containerID="b34987ce0a1fefa04c782ce4dfe7aaaae0d8f38b4711817ea271e2d96088e65f" Oct 30 07:23:43.478269 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:43.478249 4951 scope.go:122] "RemoveContainer" containerID="1cd7098c17f3398406adb9999819c2ead1525e521622cc70fa526329f51984e4" Oct 30 07:23:43.478598 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:23:43.478561 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd7098c17f3398406adb9999819c2ead1525e521622cc70fa526329f51984e4\": container with ID starting with 1cd7098c17f3398406adb9999819c2ead1525e521622cc70fa526329f51984e4 not found: ID does not exist" containerID="1cd7098c17f3398406adb9999819c2ead1525e521622cc70fa526329f51984e4" Oct 30 07:23:43.478701 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:43.478609 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd7098c17f3398406adb9999819c2ead1525e521622cc70fa526329f51984e4"} err="failed to get container status \"1cd7098c17f3398406adb9999819c2ead1525e521622cc70fa526329f51984e4\": rpc error: code = NotFound desc = could not find container \"1cd7098c17f3398406adb9999819c2ead1525e521622cc70fa526329f51984e4\": container with ID starting with 1cd7098c17f3398406adb9999819c2ead1525e521622cc70fa526329f51984e4 not found: ID does not exist" Oct 30 07:23:43.478701 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:43.478633 4951 scope.go:122] "RemoveContainer" containerID="b34987ce0a1fefa04c782ce4dfe7aaaae0d8f38b4711817ea271e2d96088e65f" Oct 30 07:23:43.478969 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:23:43.478948 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b34987ce0a1fefa04c782ce4dfe7aaaae0d8f38b4711817ea271e2d96088e65f\": container with ID starting with b34987ce0a1fefa04c782ce4dfe7aaaae0d8f38b4711817ea271e2d96088e65f not found: ID does not exist" containerID="b34987ce0a1fefa04c782ce4dfe7aaaae0d8f38b4711817ea271e2d96088e65f" Oct 30 07:23:43.479010 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:43.478975 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b34987ce0a1fefa04c782ce4dfe7aaaae0d8f38b4711817ea271e2d96088e65f"} err="failed to get container status \"b34987ce0a1fefa04c782ce4dfe7aaaae0d8f38b4711817ea271e2d96088e65f\": rpc error: code = NotFound desc = could not find container \"b34987ce0a1fefa04c782ce4dfe7aaaae0d8f38b4711817ea271e2d96088e65f\": container with ID starting with b34987ce0a1fefa04c782ce4dfe7aaaae0d8f38b4711817ea271e2d96088e65f not found: ID does not exist" Oct 30 07:23:43.502633 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:43.502604 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx"] Oct 30 07:23:43.531597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:43.528317 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-ab3d83-openshift-default-84bcb8b969-bvfcx"] Oct 30 07:23:43.808592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:43.802007 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c51720b-1e0b-4cb0-a1e2-08065c91acde" path="/var/lib/kubelet/pods/0c51720b-1e0b-4cb0-a1e2-08065c91acde/volumes" Oct 30 07:23:48.804190 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:48.804158 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh"] Oct 30 07:23:48.804840 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:48.804824 4951 cpu_manager.go:414] "containerMap: removing container" podUID="0c51720b-1e0b-4cb0-a1e2-08065c91acde" containerName="istio-proxy" Oct 30 07:23:48.804923 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:48.804843 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c51720b-1e0b-4cb0-a1e2-08065c91acde" containerName="istio-proxy" Oct 30 07:23:48.804923 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:48.804895 4951 cpu_manager.go:414] "containerMap: removing container" podUID="0c51720b-1e0b-4cb0-a1e2-08065c91acde" containerName="istio-proxy" Oct 30 07:23:48.804923 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:48.804901 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c51720b-1e0b-4cb0-a1e2-08065c91acde" containerName="istio-proxy" Oct 30 07:23:48.805099 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:48.805035 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c51720b-1e0b-4cb0-a1e2-08065c91acde" containerName="istio-proxy" Oct 30 07:23:48.826922 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:48.826866 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:23:48.865608 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:48.854810 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh"] Oct 30 07:23:48.865608 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:48.859684 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-3a5866-openshift-default-dockercfg-zrt2j\"" Oct 30 07:23:48.902450 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:48.902400 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-workload-certs\") pod \"e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:23:48.902649 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:48.902550 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b256cca-f09b-4a29-969d-da93436195ac-istiod-ca-cert\") pod \"e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:23:48.902780 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:48.902650 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-workload-socket\") pod \"e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:23:48.902780 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:48.902708 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b256cca-f09b-4a29-969d-da93436195ac-istio-podinfo\") pod \"e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:23:48.902902 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:48.902823 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-istio-data\") pod \"e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:23:48.902975 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:48.902948 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-istio-envoy\") pod \"e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:23:48.903041 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:48.903011 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b256cca-f09b-4a29-969d-da93436195ac-istio-token\") pod \"e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:23:48.903173 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:48.903150 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-credential-socket\") pod \"e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:23:48.903295 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:48.903278 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94j8f\" (UniqueName: \"kubernetes.io/projected/9b256cca-f09b-4a29-969d-da93436195ac-kube-api-access-94j8f\") pod \"e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:23:49.029316 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:49.028159 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-credential-socket\") pod \"e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:23:49.029316 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:49.028264 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94j8f\" (UniqueName: \"kubernetes.io/projected/9b256cca-f09b-4a29-969d-da93436195ac-kube-api-access-94j8f\") pod \"e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:23:49.029316 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:49.028321 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-workload-certs\") pod \"e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:23:49.029316 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:49.028381 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b256cca-f09b-4a29-969d-da93436195ac-istiod-ca-cert\") pod \"e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:23:49.029316 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:49.028412 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-workload-socket\") pod \"e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:23:49.029316 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:49.028464 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b256cca-f09b-4a29-969d-da93436195ac-istio-podinfo\") pod \"e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:23:49.029316 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:49.028557 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-istio-data\") pod \"e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:23:49.029316 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:49.028657 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-istio-envoy\") pod \"e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:23:49.029316 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:49.028699 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b256cca-f09b-4a29-969d-da93436195ac-istio-token\") pod \"e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:23:49.029951 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:49.029473 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-credential-socket\") pod \"e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:23:49.054646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:49.042859 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-workload-socket\") pod \"e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:23:49.054646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:49.043129 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-workload-certs\") pod \"e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:23:49.054646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:49.043776 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b256cca-f09b-4a29-969d-da93436195ac-istiod-ca-cert\") pod \"e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:23:49.054646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:49.044026 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-istio-data\") pod \"e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:23:49.081211 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:49.077374 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-istio-envoy\") pod \"e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:23:49.081390 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:49.081301 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b256cca-f09b-4a29-969d-da93436195ac-istio-token\") pod \"e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:23:49.104301 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:49.104243 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b256cca-f09b-4a29-969d-da93436195ac-istio-podinfo\") pod \"e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:23:49.132684 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:49.123320 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94j8f\" (UniqueName: \"kubernetes.io/projected/9b256cca-f09b-4a29-969d-da93436195ac-kube-api-access-94j8f\") pod \"e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:23:49.185669 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:49.175970 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:23:49.388702 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:49.388471 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-866967ccd7-hnk4l" Oct 30 07:23:49.593654 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:49.593624 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh"] Oct 30 07:23:49.595150 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:23:49.595114 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b256cca_f09b_4a29_969d_da93436195ac.slice/crio-112ca14aed43fd3f0af671b008a9cd1c75c9cd6120bd08463468938fab8d407a WatchSource:0}: Error finding container 112ca14aed43fd3f0af671b008a9cd1c75c9cd6120bd08463468938fab8d407a: Status 404 returned error can't find the container with id 112ca14aed43fd3f0af671b008a9cd1c75c9cd6120bd08463468938fab8d407a Oct 30 07:23:49.598553 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:49.598515 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:23:49.598668 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:49.598635 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:23:49.598721 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:49.598679 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:23:50.403530 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:50.403499 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-7889966d58-vb8fv" Oct 30 07:23:50.499874 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:50.499839 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" event={"ID":"9b256cca-f09b-4a29-969d-da93436195ac","Type":"ContainerStarted","Data":"435e10662a06f6237fe5ca36b360304526f0065f595c5d55ebaf17f58b89612d"} Oct 30 07:23:50.500073 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:50.499888 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" event={"ID":"9b256cca-f09b-4a29-969d-da93436195ac","Type":"ContainerStarted","Data":"112ca14aed43fd3f0af671b008a9cd1c75c9cd6120bd08463468938fab8d407a"} Oct 30 07:23:50.549804 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:50.546365 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" podStartSLOduration=2.546344692 podStartE2EDuration="2.546344692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:23:50.545169954 +0000 UTC m=+1636.888217954" watchObservedRunningTime="2026-10-30 07:23:50.546344692 +0000 UTC m=+1636.889392662" Oct 30 07:23:51.177272 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:51.177241 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:23:51.183848 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:51.183817 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:23:51.265647 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:51.265610 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g"] Oct 30 07:23:51.265948 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:51.265884 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" podUID="0930dc85-7926-4ea3-b498-9f88176179ca" containerName="istio-proxy" containerID="cri-o://3e4fe118c55adb34f688cb8b4c77030f018f4e4847a0f2fdb38db00f4fd7bee0" gracePeriod=30 Oct 30 07:23:51.507173 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:51.507138 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:23:51.508168 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:51.508146 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:23:51.517754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:51.517708 4951 patch_prober.go:29] interesting pod/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:23:51.517945 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:51.517761 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" podUID="d5e6118a-e82b-4c28-b716-8e6cbaa81474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:23:52.059727 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.058142 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bdd496-56b5b84df8-nh6gj"] Oct 30 07:23:52.059727 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.058448 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-56b5b84df8-nh6gj" podUID="24cd2f45-af16-4d34-b229-9b6b4c20d060" containerName="payload-processing" containerID="cri-o://54ad81fb4c272c6f58f9925b709c01f85f42f568acbbe14e9c813bc19349a03d" gracePeriod=30 Oct 30 07:23:52.084092 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.084057 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-68c9bfddc5-6vspl"] Oct 30 07:23:52.084333 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.084303 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-68c9bfddc5-6vspl" podUID="a5b46a24-504c-4849-ac82-2591ca1f0ce3" containerName="payload-pre-processing" containerID="cri-o://fb91dbce3eeb785099b66bc0e3305d7b3a9e533770c1b875adf74c65abd4139e" gracePeriod=30 Oct 30 07:23:52.234861 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.233690 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-68c9bfddc5-6vspl" podUID="a5b46a24-504c-4849-ac82-2591ca1f0ce3" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.147:9004: connect: connection refused" Oct 30 07:23:52.234861 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.233772 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-56b5b84df8-nh6gj" podUID="24cd2f45-af16-4d34-b229-9b6b4c20d060" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.148:9004: connect: connection refused" Oct 30 07:23:52.432490 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.432462 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-56b5b84df8-nh6gj" Oct 30 07:23:52.598237 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.597790 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-68c9bfddc5-6vspl" Oct 30 07:23:52.600863 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.600528 4951 generic.go:356] "Generic (PLEG): container finished" podID="a5b46a24-504c-4849-ac82-2591ca1f0ce3" containerID="fb91dbce3eeb785099b66bc0e3305d7b3a9e533770c1b875adf74c65abd4139e" exitCode=0 Oct 30 07:23:52.600863 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.600726 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-68c9bfddc5-6vspl" event={"ID":"a5b46a24-504c-4849-ac82-2591ca1f0ce3","Type":"ContainerDied","Data":"fb91dbce3eeb785099b66bc0e3305d7b3a9e533770c1b875adf74c65abd4139e"} Oct 30 07:23:52.600863 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.600788 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-68c9bfddc5-6vspl" event={"ID":"a5b46a24-504c-4849-ac82-2591ca1f0ce3","Type":"ContainerDied","Data":"dbcb2c6712c6d45056af7167e1b4427f6503629ce50df7f635b7f73f692a2977"} Oct 30 07:23:52.600863 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.600821 4951 scope.go:122] "RemoveContainer" containerID="fb91dbce3eeb785099b66bc0e3305d7b3a9e533770c1b875adf74c65abd4139e" Oct 30 07:23:52.610615 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.605440 4951 generic.go:356] "Generic (PLEG): container finished" podID="d5e6118a-e82b-4c28-b716-8e6cbaa81474" containerID="37f97d4e834152adcfa2b17a2477915d3bd49fafd25c081b312b763b8cd8905b" exitCode=0 Oct 30 07:23:52.610615 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.605656 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" event={"ID":"d5e6118a-e82b-4c28-b716-8e6cbaa81474","Type":"ContainerDied","Data":"37f97d4e834152adcfa2b17a2477915d3bd49fafd25c081b312b763b8cd8905b"} Oct 30 07:23:52.610615 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.606164 4951 scope.go:122] "RemoveContainer" containerID="37f97d4e834152adcfa2b17a2477915d3bd49fafd25c081b312b763b8cd8905b" Oct 30 07:23:52.610615 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.608615 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:23:52.610615 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.608709 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:23:52.610615 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.608753 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:23:52.650539 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.650006 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24cd2f45-af16-4d34-b229-9b6b4c20d060-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24cd2f45-af16-4d34-b229-9b6b4c20d060-plugins-config-volume\") pod \"24cd2f45-af16-4d34-b229-9b6b4c20d060\" (UID: \"24cd2f45-af16-4d34-b229-9b6b4c20d060\") " Oct 30 07:23:52.650539 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.650035 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-56b5b84df8-nh6gj" event={"ID":"24cd2f45-af16-4d34-b229-9b6b4c20d060","Type":"ContainerDied","Data":"54ad81fb4c272c6f58f9925b709c01f85f42f568acbbe14e9c813bc19349a03d"} Oct 30 07:23:52.650539 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.650158 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-56b5b84df8-nh6gj" Oct 30 07:23:52.650539 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.650274 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24cd2f45-af16-4d34-b229-9b6b4c20d060-kube-api-access-jxjs4\" (UniqueName: \"kubernetes.io/projected/24cd2f45-af16-4d34-b229-9b6b4c20d060-kube-api-access-jxjs4\") pod \"24cd2f45-af16-4d34-b229-9b6b4c20d060\" (UID: \"24cd2f45-af16-4d34-b229-9b6b4c20d060\") " Oct 30 07:23:52.651136 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.651092 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24cd2f45-af16-4d34-b229-9b6b4c20d060-plugins-config-volume" pod "24cd2f45-af16-4d34-b229-9b6b4c20d060" (UID: "24cd2f45-af16-4d34-b229-9b6b4c20d060"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:23:52.652702 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.650007 4951 generic.go:356] "Generic (PLEG): container finished" podID="24cd2f45-af16-4d34-b229-9b6b4c20d060" containerID="54ad81fb4c272c6f58f9925b709c01f85f42f568acbbe14e9c813bc19349a03d" exitCode=0 Oct 30 07:23:52.653606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.652816 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bdd496-56b5b84df8-nh6gj" event={"ID":"24cd2f45-af16-4d34-b229-9b6b4c20d060","Type":"ContainerDied","Data":"afd26f7765b949cc2217fabefbdc40bb35848aea33f1c3f6a41d818986a0ea64"} Oct 30 07:23:52.653762 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.653732 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24cd2f45-af16-4d34-b229-9b6b4c20d060-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:52.655715 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.654858 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24cd2f45-af16-4d34-b229-9b6b4c20d060-kube-api-access-jxjs4" pod "24cd2f45-af16-4d34-b229-9b6b4c20d060" (UID: "24cd2f45-af16-4d34-b229-9b6b4c20d060"). InnerVolumeSpecName "kube-api-access-jxjs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:23:52.666253 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.666134 4951 scope.go:122] "RemoveContainer" containerID="fb91dbce3eeb785099b66bc0e3305d7b3a9e533770c1b875adf74c65abd4139e" Oct 30 07:23:52.669006 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:23:52.668935 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb91dbce3eeb785099b66bc0e3305d7b3a9e533770c1b875adf74c65abd4139e\": container with ID starting with fb91dbce3eeb785099b66bc0e3305d7b3a9e533770c1b875adf74c65abd4139e not found: ID does not exist" containerID="fb91dbce3eeb785099b66bc0e3305d7b3a9e533770c1b875adf74c65abd4139e" Oct 30 07:23:52.669134 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.669018 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb91dbce3eeb785099b66bc0e3305d7b3a9e533770c1b875adf74c65abd4139e"} err="failed to get container status \"fb91dbce3eeb785099b66bc0e3305d7b3a9e533770c1b875adf74c65abd4139e\": rpc error: code = NotFound desc = could not find container \"fb91dbce3eeb785099b66bc0e3305d7b3a9e533770c1b875adf74c65abd4139e\": container with ID starting with fb91dbce3eeb785099b66bc0e3305d7b3a9e533770c1b875adf74c65abd4139e not found: ID does not exist" Oct 30 07:23:52.669134 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.669048 4951 scope.go:122] "RemoveContainer" containerID="54ad81fb4c272c6f58f9925b709c01f85f42f568acbbe14e9c813bc19349a03d" Oct 30 07:23:52.690109 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.688907 4951 scope.go:122] "RemoveContainer" containerID="54ad81fb4c272c6f58f9925b709c01f85f42f568acbbe14e9c813bc19349a03d" Oct 30 07:23:52.690109 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:23:52.689824 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54ad81fb4c272c6f58f9925b709c01f85f42f568acbbe14e9c813bc19349a03d\": container with ID starting with 54ad81fb4c272c6f58f9925b709c01f85f42f568acbbe14e9c813bc19349a03d not found: ID does not exist" containerID="54ad81fb4c272c6f58f9925b709c01f85f42f568acbbe14e9c813bc19349a03d" Oct 30 07:23:52.690109 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.689863 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54ad81fb4c272c6f58f9925b709c01f85f42f568acbbe14e9c813bc19349a03d"} err="failed to get container status \"54ad81fb4c272c6f58f9925b709c01f85f42f568acbbe14e9c813bc19349a03d\": rpc error: code = NotFound desc = could not find container \"54ad81fb4c272c6f58f9925b709c01f85f42f568acbbe14e9c813bc19349a03d\": container with ID starting with 54ad81fb4c272c6f58f9925b709c01f85f42f568acbbe14e9c813bc19349a03d not found: ID does not exist" Oct 30 07:23:52.754358 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.754319 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5b46a24-504c-4849-ac82-2591ca1f0ce3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5b46a24-504c-4849-ac82-2591ca1f0ce3-plugins-config-volume\") pod \"a5b46a24-504c-4849-ac82-2591ca1f0ce3\" (UID: \"a5b46a24-504c-4849-ac82-2591ca1f0ce3\") " Oct 30 07:23:52.754502 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.754486 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5b46a24-504c-4849-ac82-2591ca1f0ce3-kube-api-access-nwg9z\" (UniqueName: \"kubernetes.io/projected/a5b46a24-504c-4849-ac82-2591ca1f0ce3-kube-api-access-nwg9z\") pod \"a5b46a24-504c-4849-ac82-2591ca1f0ce3\" (UID: \"a5b46a24-504c-4849-ac82-2591ca1f0ce3\") " Oct 30 07:23:52.755199 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.755175 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxjs4\" (UniqueName: \"kubernetes.io/projected/24cd2f45-af16-4d34-b229-9b6b4c20d060-kube-api-access-jxjs4\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:52.755679 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.755655 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5b46a24-504c-4849-ac82-2591ca1f0ce3-plugins-config-volume" pod "a5b46a24-504c-4849-ac82-2591ca1f0ce3" (UID: "a5b46a24-504c-4849-ac82-2591ca1f0ce3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:23:52.757844 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.757808 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5b46a24-504c-4849-ac82-2591ca1f0ce3-kube-api-access-nwg9z" pod "a5b46a24-504c-4849-ac82-2591ca1f0ce3" (UID: "a5b46a24-504c-4849-ac82-2591ca1f0ce3"). InnerVolumeSpecName "kube-api-access-nwg9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:23:52.884652 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.881818 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5b46a24-504c-4849-ac82-2591ca1f0ce3-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:52.884652 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:52.881865 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwg9z\" (UniqueName: \"kubernetes.io/projected/a5b46a24-504c-4849-ac82-2591ca1f0ce3-kube-api-access-nwg9z\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:53.053301 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:53.053259 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bdd496-56b5b84df8-nh6gj"] Oct 30 07:23:53.053534 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:53.053514 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bdd496-56b5b84df8-nh6gj"] Oct 30 07:23:53.078812 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:23:53.078772 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:23:53.661816 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:53.661787 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-68c9bfddc5-6vspl" Oct 30 07:23:53.664191 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:53.664150 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" event={"ID":"d5e6118a-e82b-4c28-b716-8e6cbaa81474","Type":"ContainerStarted","Data":"d4b39ef3fcb982b9c6046669228c3ab624c29fa877d6b6c6c46f3ba5fdc126bb"} Oct 30 07:23:53.714288 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:53.714251 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-68c9bfddc5-6vspl"] Oct 30 07:23:53.738598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:53.738463 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bdd496-68c9bfddc5-6vspl"] Oct 30 07:23:53.775404 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:53.775362 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24cd2f45-af16-4d34-b229-9b6b4c20d060" path="/var/lib/kubelet/pods/24cd2f45-af16-4d34-b229-9b6b4c20d060/volumes" Oct 30 07:23:53.776042 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:53.776019 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5b46a24-504c-4849-ac82-2591ca1f0ce3" path="/var/lib/kubelet/pods/a5b46a24-504c-4849-ac82-2591ca1f0ce3/volumes" Oct 30 07:23:53.814543 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:53.814502 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:23:53.814842 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:53.814821 4951 patch_prober.go:29] interesting pod/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 07:23:53.814901 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:53.814871 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" podUID="d5e6118a-e82b-4c28-b716-8e6cbaa81474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 07:23:54.815004 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:54.814956 4951 patch_prober.go:29] interesting pod/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 07:23:54.815398 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:54.815033 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" podUID="d5e6118a-e82b-4c28-b716-8e6cbaa81474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 07:23:55.005602 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:55.005540 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3a5866-546b46c8f5-89tgg"] Oct 30 07:23:55.006226 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:55.006211 4951 cpu_manager.go:414] "containerMap: removing container" podUID="24cd2f45-af16-4d34-b229-9b6b4c20d060" containerName="payload-processing" Oct 30 07:23:55.006283 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:55.006227 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24cd2f45-af16-4d34-b229-9b6b4c20d060" containerName="payload-processing" Oct 30 07:23:55.006283 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:55.006257 4951 cpu_manager.go:414] "containerMap: removing container" podUID="a5b46a24-504c-4849-ac82-2591ca1f0ce3" containerName="payload-pre-processing" Oct 30 07:23:55.006283 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:55.006264 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5b46a24-504c-4849-ac82-2591ca1f0ce3" containerName="payload-pre-processing" Oct 30 07:23:55.006389 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:55.006380 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="24cd2f45-af16-4d34-b229-9b6b4c20d060" containerName="payload-processing" Oct 30 07:23:55.006427 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:55.006397 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5b46a24-504c-4849-ac82-2591ca1f0ce3" containerName="payload-pre-processing" Oct 30 07:23:55.006427 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:55.006406 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c51720b-1e0b-4cb0-a1e2-08065c91acde" containerName="istio-proxy" Oct 30 07:23:55.009596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:55.009549 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a5866-546b46c8f5-89tgg" Oct 30 07:23:55.039611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:55.037909 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-3a5866-dockercfg-lk4fd\"" Oct 30 07:23:55.058555 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:55.058514 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3a5866-546b46c8f5-89tgg"] Oct 30 07:23:55.080152 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:55.079085 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-3a5866\"" Oct 30 07:23:55.152605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:55.146469 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0724f769-9a02-479b-b4f0-de7f37a72e99-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3a5866-546b46c8f5-89tgg\" (UID: \"0724f769-9a02-479b-b4f0-de7f37a72e99\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a5866-546b46c8f5-89tgg" Oct 30 07:23:55.152605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:55.146670 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwjl4\" (UniqueName: \"kubernetes.io/projected/0724f769-9a02-479b-b4f0-de7f37a72e99-kube-api-access-rwjl4\") pod \"payload-pre-processing-e2e-praxis-b-3a5866-546b46c8f5-89tgg\" (UID: \"0724f769-9a02-479b-b4f0-de7f37a72e99\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a5866-546b46c8f5-89tgg" Oct 30 07:23:55.264901 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:55.264422 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwjl4\" (UniqueName: \"kubernetes.io/projected/0724f769-9a02-479b-b4f0-de7f37a72e99-kube-api-access-rwjl4\") pod \"payload-pre-processing-e2e-praxis-b-3a5866-546b46c8f5-89tgg\" (UID: \"0724f769-9a02-479b-b4f0-de7f37a72e99\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a5866-546b46c8f5-89tgg" Oct 30 07:23:55.264901 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:55.264734 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0724f769-9a02-479b-b4f0-de7f37a72e99-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3a5866-546b46c8f5-89tgg\" (UID: \"0724f769-9a02-479b-b4f0-de7f37a72e99\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a5866-546b46c8f5-89tgg" Oct 30 07:23:55.358634 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:55.354077 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0724f769-9a02-479b-b4f0-de7f37a72e99-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3a5866-546b46c8f5-89tgg\" (UID: \"0724f769-9a02-479b-b4f0-de7f37a72e99\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a5866-546b46c8f5-89tgg" Oct 30 07:23:55.389599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:55.389248 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwjl4\" (UniqueName: \"kubernetes.io/projected/0724f769-9a02-479b-b4f0-de7f37a72e99-kube-api-access-rwjl4\") pod \"payload-pre-processing-e2e-praxis-b-3a5866-546b46c8f5-89tgg\" (UID: \"0724f769-9a02-479b-b4f0-de7f37a72e99\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a5866-546b46c8f5-89tgg" Oct 30 07:23:55.646896 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:55.646809 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a5866-546b46c8f5-89tgg" Oct 30 07:23:55.814314 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:55.814279 4951 patch_prober.go:29] interesting pod/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 07:23:55.814511 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:55.814336 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" podUID="d5e6118a-e82b-4c28-b716-8e6cbaa81474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 07:23:55.974407 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:55.974380 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3a5866-546b46c8f5-89tgg"] Oct 30 07:23:55.975216 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:23:55.975192 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0724f769_9a02_479b_b4f0_de7f37a72e99.slice/crio-9ff79e1a3ee07c1d021793accc30f7c718ab414b91c7b37f2ef1048b953da330 WatchSource:0}: Error finding container 9ff79e1a3ee07c1d021793accc30f7c718ab414b91c7b37f2ef1048b953da330: Status 404 returned error can't find the container with id 9ff79e1a3ee07c1d021793accc30f7c718ab414b91c7b37f2ef1048b953da330 Oct 30 07:23:56.749145 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.748775 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:23:56.760618 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.750420 4951 generic.go:356] "Generic (PLEG): container finished" podID="0930dc85-7926-4ea3-b498-9f88176179ca" containerID="3e4fe118c55adb34f688cb8b4c77030f018f4e4847a0f2fdb38db00f4fd7bee0" exitCode=0 Oct 30 07:23:56.760618 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.750552 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" event={"ID":"0930dc85-7926-4ea3-b498-9f88176179ca","Type":"ContainerDied","Data":"3e4fe118c55adb34f688cb8b4c77030f018f4e4847a0f2fdb38db00f4fd7bee0"} Oct 30 07:23:56.761182 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.760964 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" event={"ID":"0930dc85-7926-4ea3-b498-9f88176179ca","Type":"ContainerDied","Data":"aa52395fef9d5a89b565a641bcc5f9013d6cfc04296b23dbbc469e569a6afacc"} Oct 30 07:23:56.761182 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.761030 4951 scope.go:122] "RemoveContainer" containerID="3e4fe118c55adb34f688cb8b4c77030f018f4e4847a0f2fdb38db00f4fd7bee0" Oct 30 07:23:56.766212 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.766189 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a5866-546b46c8f5-89tgg" event={"ID":"0724f769-9a02-479b-b4f0-de7f37a72e99","Type":"ContainerStarted","Data":"3730bfd3761ed7b788b27c1ac17aba5a4227467508fba773b5de6008116c1c18"} Oct 30 07:23:56.766312 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.766227 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a5866-546b46c8f5-89tgg" event={"ID":"0724f769-9a02-479b-b4f0-de7f37a72e99","Type":"ContainerStarted","Data":"9ff79e1a3ee07c1d021793accc30f7c718ab414b91c7b37f2ef1048b953da330"} Oct 30 07:23:56.766810 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.766793 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a5866-546b46c8f5-89tgg" Oct 30 07:23:56.774500 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.774483 4951 scope.go:122] "RemoveContainer" containerID="3e4fe118c55adb34f688cb8b4c77030f018f4e4847a0f2fdb38db00f4fd7bee0" Oct 30 07:23:56.774803 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:23:56.774762 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e4fe118c55adb34f688cb8b4c77030f018f4e4847a0f2fdb38db00f4fd7bee0\": container with ID starting with 3e4fe118c55adb34f688cb8b4c77030f018f4e4847a0f2fdb38db00f4fd7bee0 not found: ID does not exist" containerID="3e4fe118c55adb34f688cb8b4c77030f018f4e4847a0f2fdb38db00f4fd7bee0" Oct 30 07:23:56.774938 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.774904 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e4fe118c55adb34f688cb8b4c77030f018f4e4847a0f2fdb38db00f4fd7bee0"} err="failed to get container status \"3e4fe118c55adb34f688cb8b4c77030f018f4e4847a0f2fdb38db00f4fd7bee0\": rpc error: code = NotFound desc = could not find container \"3e4fe118c55adb34f688cb8b4c77030f018f4e4847a0f2fdb38db00f4fd7bee0\": container with ID starting with 3e4fe118c55adb34f688cb8b4c77030f018f4e4847a0f2fdb38db00f4fd7bee0 not found: ID does not exist" Oct 30 07:23:56.814681 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.814651 4951 patch_prober.go:29] interesting pod/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 07:23:56.814783 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.814708 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" podUID="d5e6118a-e82b-4c28-b716-8e6cbaa81474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 07:23:56.850100 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.847631 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-istio-envoy\") pod \"0930dc85-7926-4ea3-b498-9f88176179ca\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " Oct 30 07:23:56.853727 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.853697 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0930dc85-7926-4ea3-b498-9f88176179ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0930dc85-7926-4ea3-b498-9f88176179ca-istiod-ca-cert\") pod \"0930dc85-7926-4ea3-b498-9f88176179ca\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " Oct 30 07:23:56.854088 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.853802 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-credential-socket\") pod \"0930dc85-7926-4ea3-b498-9f88176179ca\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " Oct 30 07:23:56.854088 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.853901 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-istio-data\") pod \"0930dc85-7926-4ea3-b498-9f88176179ca\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " Oct 30 07:23:56.854088 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.853984 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0930dc85-7926-4ea3-b498-9f88176179ca-istio-token\" (UniqueName: \"kubernetes.io/projected/0930dc85-7926-4ea3-b498-9f88176179ca-istio-token\") pod \"0930dc85-7926-4ea3-b498-9f88176179ca\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " Oct 30 07:23:56.854088 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.854055 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0930dc85-7926-4ea3-b498-9f88176179ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0930dc85-7926-4ea3-b498-9f88176179ca-istio-podinfo\") pod \"0930dc85-7926-4ea3-b498-9f88176179ca\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " Oct 30 07:23:56.854415 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.854137 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0930dc85-7926-4ea3-b498-9f88176179ca-kube-api-access-g6rbq\" (UniqueName: \"kubernetes.io/projected/0930dc85-7926-4ea3-b498-9f88176179ca-kube-api-access-g6rbq\") pod \"0930dc85-7926-4ea3-b498-9f88176179ca\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " Oct 30 07:23:56.854415 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.854212 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-workload-socket\") pod \"0930dc85-7926-4ea3-b498-9f88176179ca\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " Oct 30 07:23:56.854415 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.854260 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-workload-certs\") pod \"0930dc85-7926-4ea3-b498-9f88176179ca\" (UID: \"0930dc85-7926-4ea3-b498-9f88176179ca\") " Oct 30 07:23:56.855135 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.854867 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-workload-certs" pod "0930dc85-7926-4ea3-b498-9f88176179ca" (UID: "0930dc85-7926-4ea3-b498-9f88176179ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:56.855135 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.854884 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-workload-socket" pod "0930dc85-7926-4ea3-b498-9f88176179ca" (UID: "0930dc85-7926-4ea3-b498-9f88176179ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:56.855135 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.855097 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-credential-socket" pod "0930dc85-7926-4ea3-b498-9f88176179ca" (UID: "0930dc85-7926-4ea3-b498-9f88176179ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:56.855399 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.855374 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0930dc85-7926-4ea3-b498-9f88176179ca-istiod-ca-cert" pod "0930dc85-7926-4ea3-b498-9f88176179ca" (UID: "0930dc85-7926-4ea3-b498-9f88176179ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:23:56.855679 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.855658 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-istio-data" pod "0930dc85-7926-4ea3-b498-9f88176179ca" (UID: "0930dc85-7926-4ea3-b498-9f88176179ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:56.856818 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.856518 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:56.856818 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.856537 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:56.856818 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.856546 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0930dc85-7926-4ea3-b498-9f88176179ca-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:56.856818 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.856554 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:56.856818 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.856563 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:56.857899 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.857866 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-istio-envoy" pod "0930dc85-7926-4ea3-b498-9f88176179ca" (UID: "0930dc85-7926-4ea3-b498-9f88176179ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:56.866003 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.865977 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0930dc85-7926-4ea3-b498-9f88176179ca-istio-token" pod "0930dc85-7926-4ea3-b498-9f88176179ca" (UID: "0930dc85-7926-4ea3-b498-9f88176179ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:23:56.867701 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.867666 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0930dc85-7926-4ea3-b498-9f88176179ca-kube-api-access-g6rbq" pod "0930dc85-7926-4ea3-b498-9f88176179ca" (UID: "0930dc85-7926-4ea3-b498-9f88176179ca"). InnerVolumeSpecName "kube-api-access-g6rbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:23:56.867890 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.867734 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0930dc85-7926-4ea3-b498-9f88176179ca-istio-podinfo" pod "0930dc85-7926-4ea3-b498-9f88176179ca" (UID: "0930dc85-7926-4ea3-b498-9f88176179ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:23:56.958267 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.958234 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0930dc85-7926-4ea3-b498-9f88176179ca-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:56.958583 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.958283 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0930dc85-7926-4ea3-b498-9f88176179ca-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:56.958583 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.958293 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0930dc85-7926-4ea3-b498-9f88176179ca-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:56.958583 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:56.958302 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6rbq\" (UniqueName: \"kubernetes.io/projected/0930dc85-7926-4ea3-b498-9f88176179ca-kube-api-access-g6rbq\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:23:57.780202 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:57.780172 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g" Oct 30 07:23:57.814326 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:57.814296 4951 patch_prober.go:29] interesting pod/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 07:23:57.814467 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:57.814353 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" podUID="d5e6118a-e82b-4c28-b716-8e6cbaa81474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 07:23:57.855597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:57.853881 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a5866-546b46c8f5-89tgg" podStartSLOduration=3.853861325 podStartE2EDuration="3.853861325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:23:56.856095663 +0000 UTC m=+1643.199143642" watchObservedRunningTime="2026-10-30 07:23:57.853861325 +0000 UTC m=+1644.196909291" Oct 30 07:23:57.858364 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:57.858328 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g"] Oct 30 07:23:57.867959 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:57.867918 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pfm6g"] Oct 30 07:23:58.267908 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:58.267862 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-7fd444d44c-4m5wn"] Oct 30 07:23:58.279602 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:58.274353 4951 cpu_manager.go:414] "containerMap: removing container" podUID="0930dc85-7926-4ea3-b498-9f88176179ca" containerName="istio-proxy" Oct 30 07:23:58.298688 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:58.298655 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0930dc85-7926-4ea3-b498-9f88176179ca" containerName="istio-proxy" Oct 30 07:23:58.299236 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:58.299211 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="0930dc85-7926-4ea3-b498-9f88176179ca" containerName="istio-proxy" Oct 30 07:23:58.316730 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:58.310387 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-7fd444d44c-4m5wn"] Oct 30 07:23:58.316730 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:58.310513 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-7fd444d44c-4m5wn" Oct 30 07:23:58.342455 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:58.341852 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a8f428\"" Oct 30 07:23:58.379680 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:58.379634 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zngpm\" (UniqueName: \"kubernetes.io/projected/134d6e2c-a19a-49e2-9681-cceda422f787-kube-api-access-zngpm\") pod \"payload-pre-processing-e2e-ipp-b-a8f428-7fd444d44c-4m5wn\" (UID: \"134d6e2c-a19a-49e2-9681-cceda422f787\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-7fd444d44c-4m5wn" Oct 30 07:23:58.380144 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:58.380118 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/134d6e2c-a19a-49e2-9681-cceda422f787-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a8f428-7fd444d44c-4m5wn\" (UID: \"134d6e2c-a19a-49e2-9681-cceda422f787\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-7fd444d44c-4m5wn" Oct 30 07:23:58.413793 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:58.409836 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a8f428-dockercfg-9csfk\"" Oct 30 07:23:58.415054 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:58.415023 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8f428-769f7549d6-4cvk2"] Oct 30 07:23:58.419242 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:58.419217 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-769f7549d6-4cvk2" Oct 30 07:23:58.468183 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:58.468149 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8f428-769f7549d6-4cvk2"] Oct 30 07:23:58.482306 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:58.482271 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/134d6e2c-a19a-49e2-9681-cceda422f787-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a8f428-7fd444d44c-4m5wn\" (UID: \"134d6e2c-a19a-49e2-9681-cceda422f787\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-7fd444d44c-4m5wn" Oct 30 07:23:58.482643 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:58.482621 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v89nb\" (UniqueName: \"kubernetes.io/projected/18eba7a1-a108-4536-8740-fedd18b1d479-kube-api-access-v89nb\") pod \"payload-processing-e2e-ipp-b-a8f428-769f7549d6-4cvk2\" (UID: \"18eba7a1-a108-4536-8740-fedd18b1d479\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-769f7549d6-4cvk2" Oct 30 07:23:58.482811 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:58.482794 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zngpm\" (UniqueName: \"kubernetes.io/projected/134d6e2c-a19a-49e2-9681-cceda422f787-kube-api-access-zngpm\") pod \"payload-pre-processing-e2e-ipp-b-a8f428-7fd444d44c-4m5wn\" (UID: \"134d6e2c-a19a-49e2-9681-cceda422f787\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-7fd444d44c-4m5wn" Oct 30 07:23:58.482982 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:58.482967 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18eba7a1-a108-4536-8740-fedd18b1d479-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a8f428-769f7549d6-4cvk2\" (UID: \"18eba7a1-a108-4536-8740-fedd18b1d479\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-769f7549d6-4cvk2" Oct 30 07:23:58.483475 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:58.483439 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/134d6e2c-a19a-49e2-9681-cceda422f787-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a8f428-7fd444d44c-4m5wn\" (UID: \"134d6e2c-a19a-49e2-9681-cceda422f787\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-7fd444d44c-4m5wn" Oct 30 07:23:58.509142 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:58.509104 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zngpm\" (UniqueName: \"kubernetes.io/projected/134d6e2c-a19a-49e2-9681-cceda422f787-kube-api-access-zngpm\") pod \"payload-pre-processing-e2e-ipp-b-a8f428-7fd444d44c-4m5wn\" (UID: \"134d6e2c-a19a-49e2-9681-cceda422f787\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-7fd444d44c-4m5wn" Oct 30 07:23:58.584606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:58.584488 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v89nb\" (UniqueName: \"kubernetes.io/projected/18eba7a1-a108-4536-8740-fedd18b1d479-kube-api-access-v89nb\") pod \"payload-processing-e2e-ipp-b-a8f428-769f7549d6-4cvk2\" (UID: \"18eba7a1-a108-4536-8740-fedd18b1d479\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-769f7549d6-4cvk2" Oct 30 07:23:58.584782 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:58.584662 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18eba7a1-a108-4536-8740-fedd18b1d479-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a8f428-769f7549d6-4cvk2\" (UID: \"18eba7a1-a108-4536-8740-fedd18b1d479\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-769f7549d6-4cvk2" Oct 30 07:23:58.585414 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:58.585381 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18eba7a1-a108-4536-8740-fedd18b1d479-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a8f428-769f7549d6-4cvk2\" (UID: \"18eba7a1-a108-4536-8740-fedd18b1d479\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-769f7549d6-4cvk2" Oct 30 07:23:58.608647 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:58.608612 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v89nb\" (UniqueName: \"kubernetes.io/projected/18eba7a1-a108-4536-8740-fedd18b1d479-kube-api-access-v89nb\") pod \"payload-processing-e2e-ipp-b-a8f428-769f7549d6-4cvk2\" (UID: \"18eba7a1-a108-4536-8740-fedd18b1d479\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-769f7549d6-4cvk2" Oct 30 07:23:58.649928 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:58.649903 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-pbml8_861854e2-21ba-40a6-b1b5-e3e36bd0858f/payload-processing/0.log" Oct 30 07:23:58.655847 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:58.655810 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-7fd444d44c-4m5wn" Oct 30 07:23:58.775846 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:58.775812 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-769f7549d6-4cvk2" Oct 30 07:23:58.817079 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:58.817028 4951 patch_prober.go:29] interesting pod/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 07:23:58.817642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:58.817098 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" podUID="d5e6118a-e82b-4c28-b716-8e6cbaa81474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 07:23:58.915185 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:58.915150 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-pbml8_861854e2-21ba-40a6-b1b5-e3e36bd0858f/payload-processing/0.log" Oct 30 07:23:58.980486 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:58.980455 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-7fd444d44c-4m5wn"] Oct 30 07:23:58.981075 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:23:58.981030 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod134d6e2c_a19a_49e2_9681_cceda422f787.slice/crio-e34030075264d57ac76c5421d68992f5c1c2c843b8dfe99826adf7308fa0ac12 WatchSource:0}: Error finding container e34030075264d57ac76c5421d68992f5c1c2c843b8dfe99826adf7308fa0ac12: Status 404 returned error can't find the container with id e34030075264d57ac76c5421d68992f5c1c2c843b8dfe99826adf7308fa0ac12 Oct 30 07:23:59.134164 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:59.133698 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-597d57-7889966d58-vb8fv_4152dc51-db44-4f78-b551-3310cf6137c2/payload-processing/0.log" Oct 30 07:23:59.134164 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:59.133873 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8f428-769f7549d6-4cvk2"] Oct 30 07:23:59.772658 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:59.772627 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0930dc85-7926-4ea3-b498-9f88176179ca" path="/var/lib/kubelet/pods/0930dc85-7926-4ea3-b498-9f88176179ca/volumes" Oct 30 07:23:59.797073 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:59.797034 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-769f7549d6-4cvk2" event={"ID":"18eba7a1-a108-4536-8740-fedd18b1d479","Type":"ContainerStarted","Data":"288f738c4a6e3750be63373294672e4d7c3ce8af8dcd9d0f358bb9cc21a0bb79"} Oct 30 07:23:59.797245 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:59.797100 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-769f7549d6-4cvk2" event={"ID":"18eba7a1-a108-4536-8740-fedd18b1d479","Type":"ContainerStarted","Data":"5f259574517d97e5452f674460bb22586e07868d6821b89d27763391a51320a6"} Oct 30 07:23:59.797245 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:59.797229 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-769f7549d6-4cvk2" Oct 30 07:23:59.798908 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:59.798885 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-7fd444d44c-4m5wn" event={"ID":"134d6e2c-a19a-49e2-9681-cceda422f787","Type":"ContainerStarted","Data":"e89b283b3c0335cb46a69d6a694d0bbf54c382ddf10ec1e45803cbaf6d1932c2"} Oct 30 07:23:59.799041 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:59.798920 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-7fd444d44c-4m5wn" event={"ID":"134d6e2c-a19a-49e2-9681-cceda422f787","Type":"ContainerStarted","Data":"e34030075264d57ac76c5421d68992f5c1c2c843b8dfe99826adf7308fa0ac12"} Oct 30 07:23:59.799041 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:59.798954 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-7fd444d44c-4m5wn" Oct 30 07:23:59.818820 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:59.818793 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:23:59.819259 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:59.819083 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:23:59.819763 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:59.819744 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:23:59.879172 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:59.879108 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-769f7549d6-4cvk2" podStartSLOduration=1.8790912199999998 podStartE2EDuration="1.87909122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:23:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:23:59.877372495 +0000 UTC m=+1646.220420459" watchObservedRunningTime="2026-10-30 07:23:59.87909122 +0000 UTC m=+1646.222139185" Oct 30 07:23:59.977228 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:23:59.977171 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-7fd444d44c-4m5wn" podStartSLOduration=1.977143511 podStartE2EDuration="1.977143511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:23:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:23:59.971672318 +0000 UTC m=+1646.314720283" watchObservedRunningTime="2026-10-30 07:23:59.977143511 +0000 UTC m=+1646.320191480" Oct 30 07:24:05.028902 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:05.028869 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-210755-6fc746bbc9-fw9hv"] Oct 30 07:24:05.032962 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:05.032944 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-210755-6fc746bbc9-fw9hv" Oct 30 07:24:05.051206 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:05.051152 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-210755\"" Oct 30 07:24:05.055483 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:05.055453 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-210755-6fc746bbc9-fw9hv"] Oct 30 07:24:05.063621 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:05.063594 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-210755-dockercfg-s8qcs\"" Oct 30 07:24:05.114466 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:05.114416 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-210755-6b8cfcc74-jkcp6"] Oct 30 07:24:05.225364 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:05.200859 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-210755-6b8cfcc74-jkcp6"] Oct 30 07:24:05.225364 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:05.200997 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-210755-6b8cfcc74-jkcp6" Oct 30 07:24:05.338218 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:05.338017 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/434bd254-777f-40f6-9141-a4a292dc8272-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-210755-6b8cfcc74-jkcp6\" (UID: \"434bd254-777f-40f6-9141-a4a292dc8272\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-210755-6b8cfcc74-jkcp6" Oct 30 07:24:05.362476 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:05.362431 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb4r9\" (UniqueName: \"kubernetes.io/projected/434bd254-777f-40f6-9141-a4a292dc8272-kube-api-access-nb4r9\") pod \"payload-processing-e2e-praxis-b-210755-6b8cfcc74-jkcp6\" (UID: \"434bd254-777f-40f6-9141-a4a292dc8272\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-210755-6b8cfcc74-jkcp6" Oct 30 07:24:05.363189 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:05.363165 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ebb910d-c4e9-4680-bb51-cfdb40c30428-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-210755-6fc746bbc9-fw9hv\" (UID: \"5ebb910d-c4e9-4680-bb51-cfdb40c30428\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-210755-6fc746bbc9-fw9hv" Oct 30 07:24:05.363501 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:05.363447 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sm5z\" (UniqueName: \"kubernetes.io/projected/5ebb910d-c4e9-4680-bb51-cfdb40c30428-kube-api-access-9sm5z\") pod \"payload-pre-processing-e2e-praxis-b-210755-6fc746bbc9-fw9hv\" (UID: \"5ebb910d-c4e9-4680-bb51-cfdb40c30428\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-210755-6fc746bbc9-fw9hv" Oct 30 07:24:05.465239 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:05.465202 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/434bd254-777f-40f6-9141-a4a292dc8272-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-210755-6b8cfcc74-jkcp6\" (UID: \"434bd254-777f-40f6-9141-a4a292dc8272\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-210755-6b8cfcc74-jkcp6" Oct 30 07:24:05.465425 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:05.465287 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb4r9\" (UniqueName: \"kubernetes.io/projected/434bd254-777f-40f6-9141-a4a292dc8272-kube-api-access-nb4r9\") pod \"payload-processing-e2e-praxis-b-210755-6b8cfcc74-jkcp6\" (UID: \"434bd254-777f-40f6-9141-a4a292dc8272\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-210755-6b8cfcc74-jkcp6" Oct 30 07:24:05.465425 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:05.465357 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ebb910d-c4e9-4680-bb51-cfdb40c30428-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-210755-6fc746bbc9-fw9hv\" (UID: \"5ebb910d-c4e9-4680-bb51-cfdb40c30428\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-210755-6fc746bbc9-fw9hv" Oct 30 07:24:05.465425 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:05.465417 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sm5z\" (UniqueName: \"kubernetes.io/projected/5ebb910d-c4e9-4680-bb51-cfdb40c30428-kube-api-access-9sm5z\") pod \"payload-pre-processing-e2e-praxis-b-210755-6fc746bbc9-fw9hv\" (UID: \"5ebb910d-c4e9-4680-bb51-cfdb40c30428\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-210755-6fc746bbc9-fw9hv" Oct 30 07:24:05.466162 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:05.466138 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ebb910d-c4e9-4680-bb51-cfdb40c30428-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-210755-6fc746bbc9-fw9hv\" (UID: \"5ebb910d-c4e9-4680-bb51-cfdb40c30428\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-210755-6fc746bbc9-fw9hv" Oct 30 07:24:05.466249 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:05.466224 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/434bd254-777f-40f6-9141-a4a292dc8272-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-210755-6b8cfcc74-jkcp6\" (UID: \"434bd254-777f-40f6-9141-a4a292dc8272\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-210755-6b8cfcc74-jkcp6" Oct 30 07:24:05.492036 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:05.492006 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sm5z\" (UniqueName: \"kubernetes.io/projected/5ebb910d-c4e9-4680-bb51-cfdb40c30428-kube-api-access-9sm5z\") pod \"payload-pre-processing-e2e-praxis-b-210755-6fc746bbc9-fw9hv\" (UID: \"5ebb910d-c4e9-4680-bb51-cfdb40c30428\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-210755-6fc746bbc9-fw9hv" Oct 30 07:24:05.492036 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:05.492011 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb4r9\" (UniqueName: \"kubernetes.io/projected/434bd254-777f-40f6-9141-a4a292dc8272-kube-api-access-nb4r9\") pod \"payload-processing-e2e-praxis-b-210755-6b8cfcc74-jkcp6\" (UID: \"434bd254-777f-40f6-9141-a4a292dc8272\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-210755-6b8cfcc74-jkcp6" Oct 30 07:24:05.647641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:05.647545 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-210755-6fc746bbc9-fw9hv" Oct 30 07:24:05.676539 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:05.676512 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-210755-6b8cfcc74-jkcp6" Oct 30 07:24:05.974960 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:05.974934 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-210755-6fc746bbc9-fw9hv"] Oct 30 07:24:05.975183 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:24:05.975160 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ebb910d_c4e9_4680_bb51_cfdb40c30428.slice/crio-7f77e99b20bfb17e11f8eed3605338ec8eabb4ef35f3a308b327fc09646eff8c WatchSource:0}: Error finding container 7f77e99b20bfb17e11f8eed3605338ec8eabb4ef35f3a308b327fc09646eff8c: Status 404 returned error can't find the container with id 7f77e99b20bfb17e11f8eed3605338ec8eabb4ef35f3a308b327fc09646eff8c Oct 30 07:24:06.009802 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:06.009775 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-210755-6b8cfcc74-jkcp6"] Oct 30 07:24:06.012049 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:24:06.012015 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod434bd254_777f_40f6_9141_a4a292dc8272.slice/crio-f1b3cf8706dfcc61b06b72a942fcdf82b3743458c47d4b04f87f1c6aad41e5ab WatchSource:0}: Error finding container f1b3cf8706dfcc61b06b72a942fcdf82b3743458c47d4b04f87f1c6aad41e5ab: Status 404 returned error can't find the container with id f1b3cf8706dfcc61b06b72a942fcdf82b3743458c47d4b04f87f1c6aad41e5ab Oct 30 07:24:06.925328 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:06.925277 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-210755-6b8cfcc74-jkcp6" event={"ID":"434bd254-777f-40f6-9141-a4a292dc8272","Type":"ContainerStarted","Data":"750c1f23349c6d7c6cbf881354d67e6be0f29c8d2d958a4b1bc8b43077b45f71"} Oct 30 07:24:06.925898 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:06.925353 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-210755-6b8cfcc74-jkcp6" event={"ID":"434bd254-777f-40f6-9141-a4a292dc8272","Type":"ContainerStarted","Data":"f1b3cf8706dfcc61b06b72a942fcdf82b3743458c47d4b04f87f1c6aad41e5ab"} Oct 30 07:24:06.925898 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:06.925382 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-210755-6b8cfcc74-jkcp6" Oct 30 07:24:06.927444 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:06.927410 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-210755-6fc746bbc9-fw9hv" event={"ID":"5ebb910d-c4e9-4680-bb51-cfdb40c30428","Type":"ContainerStarted","Data":"92124352f76bfaa3fa82ea3cd079897a4d3bfa7d1025d2c5a51302a072e4d48a"} Oct 30 07:24:06.927593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:06.927469 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-210755-6fc746bbc9-fw9hv" event={"ID":"5ebb910d-c4e9-4680-bb51-cfdb40c30428","Type":"ContainerStarted","Data":"7f77e99b20bfb17e11f8eed3605338ec8eabb4ef35f3a308b327fc09646eff8c"} Oct 30 07:24:06.927593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:06.927502 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-210755-6fc746bbc9-fw9hv" Oct 30 07:24:06.966519 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:06.966467 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-210755-6b8cfcc74-jkcp6" podStartSLOduration=1.966453215 podStartE2EDuration="1.966453215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:24:06.965641708 +0000 UTC m=+1653.308689674" watchObservedRunningTime="2026-10-30 07:24:06.966453215 +0000 UTC m=+1653.309501179" Oct 30 07:24:07.009329 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:07.009266 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-210755-6fc746bbc9-fw9hv" podStartSLOduration=2.00924897 podStartE2EDuration="2.00924897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:24:07.006064255 +0000 UTC m=+1653.349112222" watchObservedRunningTime="2026-10-30 07:24:07.00924897 +0000 UTC m=+1653.352296930" Oct 30 07:24:08.793786 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:08.793728 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a5866-546b46c8f5-89tgg" Oct 30 07:24:10.808187 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:10.808152 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-769f7549d6-4cvk2" Oct 30 07:24:10.808686 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:10.808247 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-7fd444d44c-4m5wn" Oct 30 07:24:17.206518 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:17.206427 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2"] Oct 30 07:24:17.212982 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:17.212954 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2" Oct 30 07:24:17.246312 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:17.246251 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2"] Oct 30 07:24:17.246658 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:17.246627 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-3a5866-serving-cert\"" Oct 30 07:24:17.275813 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:17.275771 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c27d255-21e3-475c-b23a-be2a023eafa8-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2\" (UID: \"0c27d255-21e3-475c-b23a-be2a023eafa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2" Oct 30 07:24:17.277715 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:17.275857 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqx9c\" (UniqueName: \"kubernetes.io/projected/0c27d255-21e3-475c-b23a-be2a023eafa8-kube-api-access-wqx9c\") pod \"maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2\" (UID: \"0c27d255-21e3-475c-b23a-be2a023eafa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2" Oct 30 07:24:17.277715 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:17.277662 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c27d255-21e3-475c-b23a-be2a023eafa8-metrics-certs\") pod \"maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2\" (UID: \"0c27d255-21e3-475c-b23a-be2a023eafa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2" Oct 30 07:24:17.277715 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:17.277705 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c27d255-21e3-475c-b23a-be2a023eafa8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2\" (UID: \"0c27d255-21e3-475c-b23a-be2a023eafa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2" Oct 30 07:24:17.379188 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:17.379155 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c27d255-21e3-475c-b23a-be2a023eafa8-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2\" (UID: \"0c27d255-21e3-475c-b23a-be2a023eafa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2" Oct 30 07:24:17.379354 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:17.379210 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqx9c\" (UniqueName: \"kubernetes.io/projected/0c27d255-21e3-475c-b23a-be2a023eafa8-kube-api-access-wqx9c\") pod \"maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2\" (UID: \"0c27d255-21e3-475c-b23a-be2a023eafa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2" Oct 30 07:24:17.379442 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:17.379403 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c27d255-21e3-475c-b23a-be2a023eafa8-metrics-certs\") pod \"maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2\" (UID: \"0c27d255-21e3-475c-b23a-be2a023eafa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2" Oct 30 07:24:17.379442 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:17.379437 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c27d255-21e3-475c-b23a-be2a023eafa8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2\" (UID: \"0c27d255-21e3-475c-b23a-be2a023eafa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2" Oct 30 07:24:17.379907 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:17.379882 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c27d255-21e3-475c-b23a-be2a023eafa8-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2\" (UID: \"0c27d255-21e3-475c-b23a-be2a023eafa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2" Oct 30 07:24:17.381954 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:17.381929 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c27d255-21e3-475c-b23a-be2a023eafa8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2\" (UID: \"0c27d255-21e3-475c-b23a-be2a023eafa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2" Oct 30 07:24:17.382074 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:17.382055 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c27d255-21e3-475c-b23a-be2a023eafa8-metrics-certs\") pod \"maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2\" (UID: \"0c27d255-21e3-475c-b23a-be2a023eafa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2" Oct 30 07:24:17.404350 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:17.404318 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqx9c\" (UniqueName: \"kubernetes.io/projected/0c27d255-21e3-475c-b23a-be2a023eafa8-kube-api-access-wqx9c\") pod \"maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2\" (UID: \"0c27d255-21e3-475c-b23a-be2a023eafa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2" Oct 30 07:24:17.534632 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:17.534598 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2" Oct 30 07:24:17.846893 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:17.846861 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2"] Oct 30 07:24:17.847343 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:24:17.847314 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c27d255_21e3_475c_b23a_be2a023eafa8.slice/crio-c607777744c3b963d60f42301c654a974f01ef09e5b69dd111977f84e552c6f4 WatchSource:0}: Error finding container c607777744c3b963d60f42301c654a974f01ef09e5b69dd111977f84e552c6f4: Status 404 returned error can't find the container with id c607777744c3b963d60f42301c654a974f01ef09e5b69dd111977f84e552c6f4 Oct 30 07:24:17.937985 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:17.937957 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-210755-6b8cfcc74-jkcp6" Oct 30 07:24:17.938446 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:17.938428 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-210755-6fc746bbc9-fw9hv" Oct 30 07:24:18.013540 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:18.013505 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2" event={"ID":"0c27d255-21e3-475c-b23a-be2a023eafa8","Type":"ContainerStarted","Data":"c607777744c3b963d60f42301c654a974f01ef09e5b69dd111977f84e552c6f4"} Oct 30 07:24:19.022436 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:19.022399 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2" event={"ID":"0c27d255-21e3-475c-b23a-be2a023eafa8","Type":"ContainerStarted","Data":"4b2c10ecfbbce6e1b81e19574cff90cc2223b6d51e717f11926206861bdca675"} Oct 30 07:24:19.022995 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:19.022624 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2" Oct 30 07:24:19.066713 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:19.066659 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2" podStartSLOduration=1.428985708 podStartE2EDuration="2.066645622s" totalImagesPullingTime="637.659914ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:24:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:24:19.064199073 +0000 UTC m=+1665.407247039" watchObservedRunningTime="2026-10-30 07:24:19.066645622 +0000 UTC m=+1665.409693586" Oct 30 07:24:23.024715 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:24:23.024680 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:24:25.035844 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:25.035812 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2" Oct 30 07:24:26.508163 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:26.508124 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j"] Oct 30 07:24:26.508810 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:26.508383 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j" podUID="837d2e3f-a5d4-4e78-aa65-3c277a685828" containerName="maas-api" containerID="cri-o://8e078dcdb052a4fa710e48b3e9d7eb8dc71823e8a5fab45e3f45490c80df6418" gracePeriod=30 Oct 30 07:24:26.833719 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:26.833687 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j" Oct 30 07:24:26.927786 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:26.927738 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/837d2e3f-a5d4-4e78-aa65-3c277a685828-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/837d2e3f-a5d4-4e78-aa65-3c277a685828-maas-api-tls\") pod \"837d2e3f-a5d4-4e78-aa65-3c277a685828\" (UID: \"837d2e3f-a5d4-4e78-aa65-3c277a685828\") " Oct 30 07:24:26.927997 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:26.927812 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/837d2e3f-a5d4-4e78-aa65-3c277a685828-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/837d2e3f-a5d4-4e78-aa65-3c277a685828-openshift-service-ca\") pod \"837d2e3f-a5d4-4e78-aa65-3c277a685828\" (UID: \"837d2e3f-a5d4-4e78-aa65-3c277a685828\") " Oct 30 07:24:26.927997 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:26.927861 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/837d2e3f-a5d4-4e78-aa65-3c277a685828-kube-api-access-rgzbd\" (UniqueName: \"kubernetes.io/projected/837d2e3f-a5d4-4e78-aa65-3c277a685828-kube-api-access-rgzbd\") pod \"837d2e3f-a5d4-4e78-aa65-3c277a685828\" (UID: \"837d2e3f-a5d4-4e78-aa65-3c277a685828\") " Oct 30 07:24:26.928130 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:26.928001 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/837d2e3f-a5d4-4e78-aa65-3c277a685828-metrics-certs\" (UniqueName: \"kubernetes.io/secret/837d2e3f-a5d4-4e78-aa65-3c277a685828-metrics-certs\") pod \"837d2e3f-a5d4-4e78-aa65-3c277a685828\" (UID: \"837d2e3f-a5d4-4e78-aa65-3c277a685828\") " Oct 30 07:24:26.928779 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:26.928750 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/837d2e3f-a5d4-4e78-aa65-3c277a685828-openshift-service-ca" pod "837d2e3f-a5d4-4e78-aa65-3c277a685828" (UID: "837d2e3f-a5d4-4e78-aa65-3c277a685828"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:24:26.930700 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:26.930659 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/837d2e3f-a5d4-4e78-aa65-3c277a685828-maas-api-tls" pod "837d2e3f-a5d4-4e78-aa65-3c277a685828" (UID: "837d2e3f-a5d4-4e78-aa65-3c277a685828"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:24:26.930821 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:26.930747 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/837d2e3f-a5d4-4e78-aa65-3c277a685828-metrics-certs" pod "837d2e3f-a5d4-4e78-aa65-3c277a685828" (UID: "837d2e3f-a5d4-4e78-aa65-3c277a685828"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:24:26.933929 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:26.933899 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/837d2e3f-a5d4-4e78-aa65-3c277a685828-kube-api-access-rgzbd" pod "837d2e3f-a5d4-4e78-aa65-3c277a685828" (UID: "837d2e3f-a5d4-4e78-aa65-3c277a685828"). InnerVolumeSpecName "kube-api-access-rgzbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:24:27.050652 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:27.035410 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/837d2e3f-a5d4-4e78-aa65-3c277a685828-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:24:27.050652 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:27.035451 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/837d2e3f-a5d4-4e78-aa65-3c277a685828-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:24:27.050652 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:27.035473 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgzbd\" (UniqueName: \"kubernetes.io/projected/837d2e3f-a5d4-4e78-aa65-3c277a685828-kube-api-access-rgzbd\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:24:27.050652 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:27.035485 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/837d2e3f-a5d4-4e78-aa65-3c277a685828-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:24:27.138593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:27.137017 4951 generic.go:356] "Generic (PLEG): container finished" podID="837d2e3f-a5d4-4e78-aa65-3c277a685828" containerID="8e078dcdb052a4fa710e48b3e9d7eb8dc71823e8a5fab45e3f45490c80df6418" exitCode=0 Oct 30 07:24:27.138593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:27.137312 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j" event={"ID":"837d2e3f-a5d4-4e78-aa65-3c277a685828","Type":"ContainerDied","Data":"8e078dcdb052a4fa710e48b3e9d7eb8dc71823e8a5fab45e3f45490c80df6418"} Oct 30 07:24:27.138593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:27.137368 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j" event={"ID":"837d2e3f-a5d4-4e78-aa65-3c277a685828","Type":"ContainerDied","Data":"9674891a75e9111a152d3ea8b2ae4eaf096f68548720894bef8d6241167c6c31"} Oct 30 07:24:27.138593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:27.137397 4951 scope.go:122] "RemoveContainer" containerID="8e078dcdb052a4fa710e48b3e9d7eb8dc71823e8a5fab45e3f45490c80df6418" Oct 30 07:24:27.138593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:27.137601 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j" Oct 30 07:24:27.215715 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:27.215687 4951 scope.go:122] "RemoveContainer" containerID="8e078dcdb052a4fa710e48b3e9d7eb8dc71823e8a5fab45e3f45490c80df6418" Oct 30 07:24:27.216101 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:24:27.216078 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e078dcdb052a4fa710e48b3e9d7eb8dc71823e8a5fab45e3f45490c80df6418\": container with ID starting with 8e078dcdb052a4fa710e48b3e9d7eb8dc71823e8a5fab45e3f45490c80df6418 not found: ID does not exist" containerID="8e078dcdb052a4fa710e48b3e9d7eb8dc71823e8a5fab45e3f45490c80df6418" Oct 30 07:24:27.216187 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:27.216114 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e078dcdb052a4fa710e48b3e9d7eb8dc71823e8a5fab45e3f45490c80df6418"} err="failed to get container status \"8e078dcdb052a4fa710e48b3e9d7eb8dc71823e8a5fab45e3f45490c80df6418\": rpc error: code = NotFound desc = could not find container \"8e078dcdb052a4fa710e48b3e9d7eb8dc71823e8a5fab45e3f45490c80df6418\": container with ID starting with 8e078dcdb052a4fa710e48b3e9d7eb8dc71823e8a5fab45e3f45490c80df6418 not found: ID does not exist" Oct 30 07:24:27.281401 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:27.281367 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j"] Oct 30 07:24:27.309809 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:27.309723 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc546d-7c5f84b4bd-6wg9j"] Oct 30 07:24:27.779199 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:27.779165 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="837d2e3f-a5d4-4e78-aa65-3c277a685828" path="/var/lib/kubelet/pods/837d2e3f-a5d4-4e78-aa65-3c277a685828/volumes" Oct 30 07:24:34.949284 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:34.949249 4951 scope.go:122] "RemoveContainer" containerID="1714b0aebfb2a59dbead75147b9edf8e81362625c97b77313291e98c1a6089a2" Oct 30 07:24:35.470333 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:35.470277 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn"] Oct 30 07:24:35.470950 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:35.470895 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" podUID="5ed0bc5e-986d-4699-9513-053f7fe24b65" containerName="istio-proxy" containerID="cri-o://d635ac3b796d2edd565ddb1e5fa423d6e7e6ce478f15bb5201d4c4a406dbece5" gracePeriod=30 Oct 30 07:24:40.909649 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:40.909600 4951 patch_prober.go:29] interesting pod/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 07:24:40.910179 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:40.909671 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" podUID="5ed0bc5e-986d-4699-9513-053f7fe24b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 07:24:41.184800 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.184731 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:24:41.263468 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.263421 4951 generic.go:356] "Generic (PLEG): container finished" podID="5ed0bc5e-986d-4699-9513-053f7fe24b65" containerID="d635ac3b796d2edd565ddb1e5fa423d6e7e6ce478f15bb5201d4c4a406dbece5" exitCode=0 Oct 30 07:24:41.264038 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.263759 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" Oct 30 07:24:41.264567 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.263783 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" event={"ID":"5ed0bc5e-986d-4699-9513-053f7fe24b65","Type":"ContainerDied","Data":"d635ac3b796d2edd565ddb1e5fa423d6e7e6ce478f15bb5201d4c4a406dbece5"} Oct 30 07:24:41.264816 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.264796 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn" event={"ID":"5ed0bc5e-986d-4699-9513-053f7fe24b65","Type":"ContainerDied","Data":"65a1c50f6baeb046dd16c32d68a885d7cd6561ceb3282bb042828795c92cd695"} Oct 30 07:24:41.264988 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.264935 4951 scope.go:122] "RemoveContainer" containerID="d635ac3b796d2edd565ddb1e5fa423d6e7e6ce478f15bb5201d4c4a406dbece5" Oct 30 07:24:41.275863 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.275835 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ed0bc5e-986d-4699-9513-053f7fe24b65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ed0bc5e-986d-4699-9513-053f7fe24b65-istiod-ca-cert\") pod \"5ed0bc5e-986d-4699-9513-053f7fe24b65\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " Oct 30 07:24:41.275960 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.275912 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ed0bc5e-986d-4699-9513-053f7fe24b65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ed0bc5e-986d-4699-9513-053f7fe24b65-istio-podinfo\") pod \"5ed0bc5e-986d-4699-9513-053f7fe24b65\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " Oct 30 07:24:41.276125 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.276108 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-workload-socket\") pod \"5ed0bc5e-986d-4699-9513-053f7fe24b65\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " Oct 30 07:24:41.276191 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.276147 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ed0bc5e-986d-4699-9513-053f7fe24b65-kube-api-access-6h79x\" (UniqueName: \"kubernetes.io/projected/5ed0bc5e-986d-4699-9513-053f7fe24b65-kube-api-access-6h79x\") pod \"5ed0bc5e-986d-4699-9513-053f7fe24b65\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " Oct 30 07:24:41.276260 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.276244 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-istio-data\") pod \"5ed0bc5e-986d-4699-9513-053f7fe24b65\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " Oct 30 07:24:41.276325 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.276309 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-workload-certs\") pod \"5ed0bc5e-986d-4699-9513-053f7fe24b65\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " Oct 30 07:24:41.276369 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.276357 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-istio-envoy\") pod \"5ed0bc5e-986d-4699-9513-053f7fe24b65\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " Oct 30 07:24:41.276410 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.276391 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ed0bc5e-986d-4699-9513-053f7fe24b65-istio-token\" (UniqueName: \"kubernetes.io/projected/5ed0bc5e-986d-4699-9513-053f7fe24b65-istio-token\") pod \"5ed0bc5e-986d-4699-9513-053f7fe24b65\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " Oct 30 07:24:41.276472 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.276456 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-credential-socket\") pod \"5ed0bc5e-986d-4699-9513-053f7fe24b65\" (UID: \"5ed0bc5e-986d-4699-9513-053f7fe24b65\") " Oct 30 07:24:41.278040 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.278012 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-istio-data" pod "5ed0bc5e-986d-4699-9513-053f7fe24b65" (UID: "5ed0bc5e-986d-4699-9513-053f7fe24b65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:24:41.279241 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.279212 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ed0bc5e-986d-4699-9513-053f7fe24b65-kube-api-access-6h79x" pod "5ed0bc5e-986d-4699-9513-053f7fe24b65" (UID: "5ed0bc5e-986d-4699-9513-053f7fe24b65"). InnerVolumeSpecName "kube-api-access-6h79x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:24:41.280714 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.280691 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-workload-socket" pod "5ed0bc5e-986d-4699-9513-053f7fe24b65" (UID: "5ed0bc5e-986d-4699-9513-053f7fe24b65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:24:41.280867 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.280814 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ed0bc5e-986d-4699-9513-053f7fe24b65-istiod-ca-cert" pod "5ed0bc5e-986d-4699-9513-053f7fe24b65" (UID: "5ed0bc5e-986d-4699-9513-053f7fe24b65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:24:41.281141 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.281106 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-workload-certs" pod "5ed0bc5e-986d-4699-9513-053f7fe24b65" (UID: "5ed0bc5e-986d-4699-9513-053f7fe24b65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:24:41.281365 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.281345 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-credential-socket" pod "5ed0bc5e-986d-4699-9513-053f7fe24b65" (UID: "5ed0bc5e-986d-4699-9513-053f7fe24b65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:24:41.288720 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.288594 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ed0bc5e-986d-4699-9513-053f7fe24b65-istio-token" pod "5ed0bc5e-986d-4699-9513-053f7fe24b65" (UID: "5ed0bc5e-986d-4699-9513-053f7fe24b65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:24:41.288720 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.288683 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-istio-envoy" pod "5ed0bc5e-986d-4699-9513-053f7fe24b65" (UID: "5ed0bc5e-986d-4699-9513-053f7fe24b65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:24:41.289120 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.288983 4951 scope.go:122] "RemoveContainer" containerID="800de26dd48119553231546144674293d0dbf81bdbd6ecf9c935135284d2aabb" Oct 30 07:24:41.290325 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.290286 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ed0bc5e-986d-4699-9513-053f7fe24b65-istio-podinfo" pod "5ed0bc5e-986d-4699-9513-053f7fe24b65" (UID: "5ed0bc5e-986d-4699-9513-053f7fe24b65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:24:41.324625 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.324596 4951 scope.go:122] "RemoveContainer" containerID="d635ac3b796d2edd565ddb1e5fa423d6e7e6ce478f15bb5201d4c4a406dbece5" Oct 30 07:24:41.325022 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:24:41.324985 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d635ac3b796d2edd565ddb1e5fa423d6e7e6ce478f15bb5201d4c4a406dbece5\": container with ID starting with d635ac3b796d2edd565ddb1e5fa423d6e7e6ce478f15bb5201d4c4a406dbece5 not found: ID does not exist" containerID="d635ac3b796d2edd565ddb1e5fa423d6e7e6ce478f15bb5201d4c4a406dbece5" Oct 30 07:24:41.325165 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.325033 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d635ac3b796d2edd565ddb1e5fa423d6e7e6ce478f15bb5201d4c4a406dbece5"} err="failed to get container status \"d635ac3b796d2edd565ddb1e5fa423d6e7e6ce478f15bb5201d4c4a406dbece5\": rpc error: code = NotFound desc = could not find container \"d635ac3b796d2edd565ddb1e5fa423d6e7e6ce478f15bb5201d4c4a406dbece5\": container with ID starting with d635ac3b796d2edd565ddb1e5fa423d6e7e6ce478f15bb5201d4c4a406dbece5 not found: ID does not exist" Oct 30 07:24:41.325165 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.325058 4951 scope.go:122] "RemoveContainer" containerID="800de26dd48119553231546144674293d0dbf81bdbd6ecf9c935135284d2aabb" Oct 30 07:24:41.325456 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:24:41.325402 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"800de26dd48119553231546144674293d0dbf81bdbd6ecf9c935135284d2aabb\": container with ID starting with 800de26dd48119553231546144674293d0dbf81bdbd6ecf9c935135284d2aabb not found: ID does not exist" containerID="800de26dd48119553231546144674293d0dbf81bdbd6ecf9c935135284d2aabb" Oct 30 07:24:41.325635 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.325454 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"800de26dd48119553231546144674293d0dbf81bdbd6ecf9c935135284d2aabb"} err="failed to get container status \"800de26dd48119553231546144674293d0dbf81bdbd6ecf9c935135284d2aabb\": rpc error: code = NotFound desc = could not find container \"800de26dd48119553231546144674293d0dbf81bdbd6ecf9c935135284d2aabb\": container with ID starting with 800de26dd48119553231546144674293d0dbf81bdbd6ecf9c935135284d2aabb not found: ID does not exist" Oct 30 07:24:41.378374 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.378348 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:24:41.378374 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.378374 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ed0bc5e-986d-4699-9513-053f7fe24b65-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:24:41.378548 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.378385 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ed0bc5e-986d-4699-9513-053f7fe24b65-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:24:41.378548 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.378393 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:24:41.378548 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.378404 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h79x\" (UniqueName: \"kubernetes.io/projected/5ed0bc5e-986d-4699-9513-053f7fe24b65-kube-api-access-6h79x\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:24:41.378548 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.378413 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:24:41.378548 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.378421 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:24:41.378548 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.378430 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ed0bc5e-986d-4699-9513-053f7fe24b65-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:24:41.378548 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.378437 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ed0bc5e-986d-4699-9513-053f7fe24b65-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:24:41.639854 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.639824 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn"] Oct 30 07:24:41.651890 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.651821 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-fc546d-openshift-default-56db84d84c-qxgbn"] Oct 30 07:24:41.772937 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:41.772859 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ed0bc5e-986d-4699-9513-053f7fe24b65" path="/var/lib/kubelet/pods/5ed0bc5e-986d-4699-9513-053f7fe24b65/volumes" Oct 30 07:24:42.272987 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:42.272948 4951 generic.go:356] "Generic (PLEG): container finished" podID="9b256cca-f09b-4a29-969d-da93436195ac" containerID="435e10662a06f6237fe5ca36b360304526f0065f595c5d55ebaf17f58b89612d" exitCode=0 Oct 30 07:24:42.273414 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:42.273018 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" event={"ID":"9b256cca-f09b-4a29-969d-da93436195ac","Type":"ContainerDied","Data":"435e10662a06f6237fe5ca36b360304526f0065f595c5d55ebaf17f58b89612d"} Oct 30 07:24:42.273564 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:42.273545 4951 scope.go:122] "RemoveContainer" containerID="435e10662a06f6237fe5ca36b360304526f0065f595c5d55ebaf17f58b89612d" Oct 30 07:24:42.275241 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:42.275210 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:24:42.275309 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:42.275285 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:24:42.275346 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:42.275321 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:24:43.282312 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:43.282278 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" event={"ID":"9b256cca-f09b-4a29-969d-da93436195ac","Type":"ContainerStarted","Data":"b777db902e8c5b70bfdba2a1ce2ed20d2d90e98d0bae47db958c1bc2101f69c3"} Oct 30 07:24:44.176824 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:44.176785 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:24:44.177033 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:44.176977 4951 patch_prober.go:29] interesting pod/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 07:24:44.177097 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:44.177029 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" podUID="9b256cca-f09b-4a29-969d-da93436195ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 07:24:45.189658 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:45.185708 4951 patch_prober.go:29] interesting pod/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 07:24:45.189658 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:45.185788 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" podUID="9b256cca-f09b-4a29-969d-da93436195ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 07:24:46.177157 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:46.177115 4951 patch_prober.go:29] interesting pod/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 07:24:46.177355 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:46.177176 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" podUID="9b256cca-f09b-4a29-969d-da93436195ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 07:24:47.182600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:47.177964 4951 patch_prober.go:29] interesting pod/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 07:24:47.182600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:47.178026 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" podUID="9b256cca-f09b-4a29-969d-da93436195ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 07:24:48.177095 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:48.176960 4951 patch_prober.go:29] interesting pod/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 07:24:48.177095 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:48.177024 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" podUID="9b256cca-f09b-4a29-969d-da93436195ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 07:24:49.196598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:49.193950 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:24:49.196598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:49.195972 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:24:49.209082 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:24:49.209035 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:24:53.025131 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:24:53.025088 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:25:03.504939 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:03.504898 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-96ca87-65d6cb4bdd-clzvf"] Oct 30 07:25:04.747683 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:04.747645 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm"] Oct 30 07:25:04.748592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:04.748560 4951 cpu_manager.go:414] "containerMap: removing container" podUID="837d2e3f-a5d4-4e78-aa65-3c277a685828" containerName="maas-api" Oct 30 07:25:04.748664 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:04.748593 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="837d2e3f-a5d4-4e78-aa65-3c277a685828" containerName="maas-api" Oct 30 07:25:04.748664 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:04.748633 4951 cpu_manager.go:414] "containerMap: removing container" podUID="5ed0bc5e-986d-4699-9513-053f7fe24b65" containerName="istio-proxy" Oct 30 07:25:04.748664 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:04.748638 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ed0bc5e-986d-4699-9513-053f7fe24b65" containerName="istio-proxy" Oct 30 07:25:04.748664 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:04.748648 4951 cpu_manager.go:414] "containerMap: removing container" podUID="5ed0bc5e-986d-4699-9513-053f7fe24b65" containerName="istio-proxy" Oct 30 07:25:04.748664 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:04.748653 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ed0bc5e-986d-4699-9513-053f7fe24b65" containerName="istio-proxy" Oct 30 07:25:04.748869 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:04.748746 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ed0bc5e-986d-4699-9513-053f7fe24b65" containerName="istio-proxy" Oct 30 07:25:04.748869 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:04.748756 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ed0bc5e-986d-4699-9513-053f7fe24b65" containerName="istio-proxy" Oct 30 07:25:04.748869 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:04.748772 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="837d2e3f-a5d4-4e78-aa65-3c277a685828" containerName="maas-api" Oct 30 07:25:04.753271 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:04.753248 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:25:04.771084 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:04.771055 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-5bb165-openshift-default-dockercfg-cl4gm\"" Oct 30 07:25:04.790258 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:04.790216 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm"] Oct 30 07:25:04.921530 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:04.921482 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4506abe9-6f43-4070-9b7b-64631f5be137-istiod-ca-cert\") pod \"e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:25:04.925694 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:04.925656 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-credential-socket\") pod \"e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:25:04.925907 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:04.925887 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-workload-certs\") pod \"e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:25:04.934758 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:04.934723 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5dck\" (UniqueName: \"kubernetes.io/projected/4506abe9-6f43-4070-9b7b-64631f5be137-kube-api-access-x5dck\") pod \"e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:25:04.935073 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:04.935053 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-istio-envoy\") pod \"e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:25:04.935212 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:04.935197 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4506abe9-6f43-4070-9b7b-64631f5be137-istio-podinfo\") pod \"e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:25:04.935327 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:04.935312 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-workload-socket\") pod \"e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:25:04.935499 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:04.935484 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4506abe9-6f43-4070-9b7b-64631f5be137-istio-token\") pod \"e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:25:04.951677 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:04.951618 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-istio-data\") pod \"e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:25:05.056602 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:05.054428 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-istio-data\") pod \"e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:25:05.056602 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:05.054565 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4506abe9-6f43-4070-9b7b-64631f5be137-istiod-ca-cert\") pod \"e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:25:05.056602 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:05.054676 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-credential-socket\") pod \"e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:25:05.056602 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:05.054705 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-workload-certs\") pod \"e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:25:05.056602 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:05.054776 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5dck\" (UniqueName: \"kubernetes.io/projected/4506abe9-6f43-4070-9b7b-64631f5be137-kube-api-access-x5dck\") pod \"e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:25:05.056602 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:05.054870 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-istio-envoy\") pod \"e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:25:05.056602 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:05.054898 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4506abe9-6f43-4070-9b7b-64631f5be137-istio-podinfo\") pod \"e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:25:05.056602 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:05.054931 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-workload-socket\") pod \"e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:25:05.056602 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:05.055015 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4506abe9-6f43-4070-9b7b-64631f5be137-istio-token\") pod \"e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:25:05.056602 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:05.055813 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-istio-data\") pod \"e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:25:05.056602 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:05.056434 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4506abe9-6f43-4070-9b7b-64631f5be137-istiod-ca-cert\") pod \"e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:25:05.057460 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:05.056683 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-credential-socket\") pod \"e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:25:05.057460 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:05.056913 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-workload-certs\") pod \"e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:25:05.079769 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:05.078954 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-workload-socket\") pod \"e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:25:05.114280 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:05.114231 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5dck\" (UniqueName: \"kubernetes.io/projected/4506abe9-6f43-4070-9b7b-64631f5be137-kube-api-access-x5dck\") pod \"e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:25:05.147946 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:05.147915 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4506abe9-6f43-4070-9b7b-64631f5be137-istio-podinfo\") pod \"e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:25:05.148175 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:05.147965 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-istio-envoy\") pod \"e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:25:05.153616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:05.152142 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4506abe9-6f43-4070-9b7b-64631f5be137-istio-token\") pod \"e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:25:05.378436 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:05.378351 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:25:05.701136 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:05.701110 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm"] Oct 30 07:25:05.701880 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:25:05.701857 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4506abe9_6f43_4070_9b7b_64631f5be137.slice/crio-cef59306f527fb871e9e0badce78cec3996e4a8c6e53ff09f516b9c7210529eb WatchSource:0}: Error finding container cef59306f527fb871e9e0badce78cec3996e4a8c6e53ff09f516b9c7210529eb: Status 404 returned error can't find the container with id cef59306f527fb871e9e0badce78cec3996e4a8c6e53ff09f516b9c7210529eb Oct 30 07:25:05.704706 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:05.704668 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:25:05.704833 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:05.704744 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:25:05.704833 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:05.704780 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:25:06.536663 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:06.536621 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" event={"ID":"4506abe9-6f43-4070-9b7b-64631f5be137","Type":"ContainerStarted","Data":"a1054e6f9ac21300a17a382d2aaf5d3039c4ef2d281d401b5440b7fc91f77dca"} Oct 30 07:25:06.537146 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:06.536695 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" event={"ID":"4506abe9-6f43-4070-9b7b-64631f5be137","Type":"ContainerStarted","Data":"cef59306f527fb871e9e0badce78cec3996e4a8c6e53ff09f516b9c7210529eb"} Oct 30 07:25:06.596936 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:06.596882 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" podStartSLOduration=2.596868639 podStartE2EDuration="2.596868639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:06.593655423 +0000 UTC m=+1712.936703395" watchObservedRunningTime="2026-10-30 07:25:06.596868639 +0000 UTC m=+1712.939916604" Oct 30 07:25:07.378630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:07.378601 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:25:07.383869 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:07.383844 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:25:07.544281 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:07.544246 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:25:07.545219 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:07.545197 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:25:08.505398 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:08.505347 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-96ca87-65d6cb4bdd-clzvf" podUID="41a7e771-8212-4749-a905-4826e1814eff" containerName="payload-processing" containerID="cri-o://4a1cfcd5866f05cb19e0df73c4004c1ae02f096c3a218971dc3fd9b6077b51e5" gracePeriod=25 Oct 30 07:25:08.875027 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:08.874992 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-96ca87-65d6cb4bdd-clzvf" Oct 30 07:25:08.937647 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:08.937612 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41a7e771-8212-4749-a905-4826e1814eff-kube-api-access-ffhjf\" (UniqueName: \"kubernetes.io/projected/41a7e771-8212-4749-a905-4826e1814eff-kube-api-access-ffhjf\") pod \"41a7e771-8212-4749-a905-4826e1814eff\" (UID: \"41a7e771-8212-4749-a905-4826e1814eff\") " Oct 30 07:25:08.937837 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:08.937701 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41a7e771-8212-4749-a905-4826e1814eff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41a7e771-8212-4749-a905-4826e1814eff-plugins-config-volume\") pod \"41a7e771-8212-4749-a905-4826e1814eff\" (UID: \"41a7e771-8212-4749-a905-4826e1814eff\") " Oct 30 07:25:08.938962 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:08.938906 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41a7e771-8212-4749-a905-4826e1814eff-plugins-config-volume" pod "41a7e771-8212-4749-a905-4826e1814eff" (UID: "41a7e771-8212-4749-a905-4826e1814eff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:25:08.952741 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:08.952712 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a7e771-8212-4749-a905-4826e1814eff-kube-api-access-ffhjf" pod "41a7e771-8212-4749-a905-4826e1814eff" (UID: "41a7e771-8212-4749-a905-4826e1814eff"). InnerVolumeSpecName "kube-api-access-ffhjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:25:09.039686 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:09.039647 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffhjf\" (UniqueName: \"kubernetes.io/projected/41a7e771-8212-4749-a905-4826e1814eff-kube-api-access-ffhjf\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:09.039868 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:09.039719 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41a7e771-8212-4749-a905-4826e1814eff-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:09.568397 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:09.568361 4951 generic.go:356] "Generic (PLEG): container finished" podID="41a7e771-8212-4749-a905-4826e1814eff" containerID="4a1cfcd5866f05cb19e0df73c4004c1ae02f096c3a218971dc3fd9b6077b51e5" exitCode=0 Oct 30 07:25:09.568612 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:09.568412 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-96ca87-65d6cb4bdd-clzvf" Oct 30 07:25:09.568612 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:09.568455 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-96ca87-65d6cb4bdd-clzvf" event={"ID":"41a7e771-8212-4749-a905-4826e1814eff","Type":"ContainerDied","Data":"4a1cfcd5866f05cb19e0df73c4004c1ae02f096c3a218971dc3fd9b6077b51e5"} Oct 30 07:25:09.568612 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:09.568517 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-96ca87-65d6cb4bdd-clzvf" event={"ID":"41a7e771-8212-4749-a905-4826e1814eff","Type":"ContainerDied","Data":"d5c1b1f555ce52f90c6d4720634cbe9d1163f950a627228f5e675926902995c1"} Oct 30 07:25:09.568612 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:09.568539 4951 scope.go:122] "RemoveContainer" containerID="4a1cfcd5866f05cb19e0df73c4004c1ae02f096c3a218971dc3fd9b6077b51e5" Oct 30 07:25:09.580167 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:09.580148 4951 scope.go:122] "RemoveContainer" containerID="4a1cfcd5866f05cb19e0df73c4004c1ae02f096c3a218971dc3fd9b6077b51e5" Oct 30 07:25:09.580507 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:25:09.580486 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a1cfcd5866f05cb19e0df73c4004c1ae02f096c3a218971dc3fd9b6077b51e5\": container with ID starting with 4a1cfcd5866f05cb19e0df73c4004c1ae02f096c3a218971dc3fd9b6077b51e5 not found: ID does not exist" containerID="4a1cfcd5866f05cb19e0df73c4004c1ae02f096c3a218971dc3fd9b6077b51e5" Oct 30 07:25:09.580627 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:09.580514 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a1cfcd5866f05cb19e0df73c4004c1ae02f096c3a218971dc3fd9b6077b51e5"} err="failed to get container status \"4a1cfcd5866f05cb19e0df73c4004c1ae02f096c3a218971dc3fd9b6077b51e5\": rpc error: code = NotFound desc = could not find container \"4a1cfcd5866f05cb19e0df73c4004c1ae02f096c3a218971dc3fd9b6077b51e5\": container with ID starting with 4a1cfcd5866f05cb19e0df73c4004c1ae02f096c3a218971dc3fd9b6077b51e5 not found: ID does not exist" Oct 30 07:25:09.644041 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:09.644007 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-96ca87-65d6cb4bdd-clzvf"] Oct 30 07:25:09.673976 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:09.673936 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-96ca87-65d6cb4bdd-clzvf"] Oct 30 07:25:09.772977 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:09.772944 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41a7e771-8212-4749-a905-4826e1814eff" path="/var/lib/kubelet/pods/41a7e771-8212-4749-a905-4826e1814eff/volumes" Oct 30 07:25:11.362040 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:11.362003 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1beb92-8546db7586-cm9b4"] Oct 30 07:25:11.431684 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:11.431651 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1beb92-fc79c45ff-gpklb"] Oct 30 07:25:12.237593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:12.237150 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw"] Oct 30 07:25:12.237593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:12.237394 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw" podUID="e312e239-4207-479a-a33e-d7769edd58b5" containerName="maas-api" containerID="cri-o://654ebbd2ccf089d6010bd315dc78fd512df44f970b55254ac4a87e16bfb51e6d" gracePeriod=30 Oct 30 07:25:12.681777 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:12.666777 4951 generic.go:356] "Generic (PLEG): container finished" podID="e312e239-4207-479a-a33e-d7769edd58b5" containerID="654ebbd2ccf089d6010bd315dc78fd512df44f970b55254ac4a87e16bfb51e6d" exitCode=0 Oct 30 07:25:12.681777 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:12.666915 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw" event={"ID":"e312e239-4207-479a-a33e-d7769edd58b5","Type":"ContainerDied","Data":"654ebbd2ccf089d6010bd315dc78fd512df44f970b55254ac4a87e16bfb51e6d"} Oct 30 07:25:12.681777 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:12.666968 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw" event={"ID":"e312e239-4207-479a-a33e-d7769edd58b5","Type":"ContainerDied","Data":"53066f4a077c0f714183c01ff3caf21d013a6b5f40356487e5bd7f15e53472b5"} Oct 30 07:25:12.681777 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:12.666994 4951 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53066f4a077c0f714183c01ff3caf21d013a6b5f40356487e5bd7f15e53472b5" Oct 30 07:25:12.681777 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:12.672342 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw" Oct 30 07:25:12.681777 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:12.674896 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e312e239-4207-479a-a33e-d7769edd58b5-kube-api-access-gb5tk\" (UniqueName: \"kubernetes.io/projected/e312e239-4207-479a-a33e-d7769edd58b5-kube-api-access-gb5tk\") pod \"e312e239-4207-479a-a33e-d7769edd58b5\" (UID: \"e312e239-4207-479a-a33e-d7769edd58b5\") " Oct 30 07:25:12.681777 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:12.675048 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e312e239-4207-479a-a33e-d7769edd58b5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e312e239-4207-479a-a33e-d7769edd58b5-metrics-certs\") pod \"e312e239-4207-479a-a33e-d7769edd58b5\" (UID: \"e312e239-4207-479a-a33e-d7769edd58b5\") " Oct 30 07:25:12.681777 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:12.675087 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e312e239-4207-479a-a33e-d7769edd58b5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e312e239-4207-479a-a33e-d7769edd58b5-openshift-service-ca\") pod \"e312e239-4207-479a-a33e-d7769edd58b5\" (UID: \"e312e239-4207-479a-a33e-d7769edd58b5\") " Oct 30 07:25:12.681777 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:12.675222 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e312e239-4207-479a-a33e-d7769edd58b5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e312e239-4207-479a-a33e-d7769edd58b5-maas-api-tls\") pod \"e312e239-4207-479a-a33e-d7769edd58b5\" (UID: \"e312e239-4207-479a-a33e-d7769edd58b5\") " Oct 30 07:25:12.681777 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:12.677255 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e312e239-4207-479a-a33e-d7769edd58b5-openshift-service-ca" pod "e312e239-4207-479a-a33e-d7769edd58b5" (UID: "e312e239-4207-479a-a33e-d7769edd58b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:25:12.681777 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:12.681014 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e312e239-4207-479a-a33e-d7769edd58b5-kube-api-access-gb5tk" pod "e312e239-4207-479a-a33e-d7769edd58b5" (UID: "e312e239-4207-479a-a33e-d7769edd58b5"). InnerVolumeSpecName "kube-api-access-gb5tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:25:12.683688 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:12.683641 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e312e239-4207-479a-a33e-d7769edd58b5-metrics-certs" pod "e312e239-4207-479a-a33e-d7769edd58b5" (UID: "e312e239-4207-479a-a33e-d7769edd58b5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:25:12.684029 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:12.683994 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e312e239-4207-479a-a33e-d7769edd58b5-maas-api-tls" pod "e312e239-4207-479a-a33e-d7769edd58b5" (UID: "e312e239-4207-479a-a33e-d7769edd58b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:25:12.799649 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:12.790286 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e312e239-4207-479a-a33e-d7769edd58b5-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:12.799904 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:12.799880 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb5tk\" (UniqueName: \"kubernetes.io/projected/e312e239-4207-479a-a33e-d7769edd58b5-kube-api-access-gb5tk\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:12.800005 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:12.799988 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e312e239-4207-479a-a33e-d7769edd58b5-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:12.800135 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:12.800120 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e312e239-4207-479a-a33e-d7769edd58b5-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:13.673871 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:13.673794 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw" Oct 30 07:25:13.726378 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:13.726335 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw"] Oct 30 07:25:13.738947 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:13.738911 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1beb92-fc44d7d49-k5hnw"] Oct 30 07:25:13.772471 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:13.772437 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e312e239-4207-479a-a33e-d7769edd58b5" path="/var/lib/kubelet/pods/e312e239-4207-479a-a33e-d7769edd58b5/volumes" Oct 30 07:25:14.250505 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.250463 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4"] Oct 30 07:25:14.250759 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.250729 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4" podUID="4b2cc236-34ca-4719-a938-2bf84d312d6e" containerName="maas-api" containerID="cri-o://b68e9367e957f21d68572c4f64f695fe0ce03480ea8247ed894464a60710b533" gracePeriod=30 Oct 30 07:25:14.410736 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.410695 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8f428-769f7549d6-4cvk2"] Oct 30 07:25:14.410962 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.410937 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-769f7549d6-4cvk2" podUID="18eba7a1-a108-4536-8740-fedd18b1d479" containerName="payload-processing" containerID="cri-o://288f738c4a6e3750be63373294672e4d7c3ce8af8dcd9d0f358bb9cc21a0bb79" gracePeriod=30 Oct 30 07:25:14.430694 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.430665 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-7fd444d44c-4m5wn"] Oct 30 07:25:14.430932 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.430905 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-7fd444d44c-4m5wn" podUID="134d6e2c-a19a-49e2-9681-cceda422f787" containerName="payload-pre-processing" containerID="cri-o://e89b283b3c0335cb46a69d6a694d0bbf54c382ddf10ec1e45803cbaf6d1932c2" gracePeriod=30 Oct 30 07:25:14.619615 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.619357 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4" Oct 30 07:25:14.626226 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.625697 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b2cc236-34ca-4719-a938-2bf84d312d6e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b2cc236-34ca-4719-a938-2bf84d312d6e-maas-api-tls\") pod \"4b2cc236-34ca-4719-a938-2bf84d312d6e\" (UID: \"4b2cc236-34ca-4719-a938-2bf84d312d6e\") " Oct 30 07:25:14.626401 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.626373 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b2cc236-34ca-4719-a938-2bf84d312d6e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b2cc236-34ca-4719-a938-2bf84d312d6e-metrics-certs\") pod \"4b2cc236-34ca-4719-a938-2bf84d312d6e\" (UID: \"4b2cc236-34ca-4719-a938-2bf84d312d6e\") " Oct 30 07:25:14.626900 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.626606 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b2cc236-34ca-4719-a938-2bf84d312d6e-kube-api-access-nwqjm\" (UniqueName: \"kubernetes.io/projected/4b2cc236-34ca-4719-a938-2bf84d312d6e-kube-api-access-nwqjm\") pod \"4b2cc236-34ca-4719-a938-2bf84d312d6e\" (UID: \"4b2cc236-34ca-4719-a938-2bf84d312d6e\") " Oct 30 07:25:14.626900 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.626712 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b2cc236-34ca-4719-a938-2bf84d312d6e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b2cc236-34ca-4719-a938-2bf84d312d6e-openshift-service-ca\") pod \"4b2cc236-34ca-4719-a938-2bf84d312d6e\" (UID: \"4b2cc236-34ca-4719-a938-2bf84d312d6e\") " Oct 30 07:25:14.628774 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.627607 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b2cc236-34ca-4719-a938-2bf84d312d6e-openshift-service-ca" pod "4b2cc236-34ca-4719-a938-2bf84d312d6e" (UID: "4b2cc236-34ca-4719-a938-2bf84d312d6e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:25:14.631509 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.631475 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b2cc236-34ca-4719-a938-2bf84d312d6e-kube-api-access-nwqjm" pod "4b2cc236-34ca-4719-a938-2bf84d312d6e" (UID: "4b2cc236-34ca-4719-a938-2bf84d312d6e"). InnerVolumeSpecName "kube-api-access-nwqjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:25:14.634789 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.634073 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b2cc236-34ca-4719-a938-2bf84d312d6e-maas-api-tls" pod "4b2cc236-34ca-4719-a938-2bf84d312d6e" (UID: "4b2cc236-34ca-4719-a938-2bf84d312d6e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:25:14.647706 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.647653 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b2cc236-34ca-4719-a938-2bf84d312d6e-metrics-certs" pod "4b2cc236-34ca-4719-a938-2bf84d312d6e" (UID: "4b2cc236-34ca-4719-a938-2bf84d312d6e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:25:14.687852 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.687819 4951 generic.go:356] "Generic (PLEG): container finished" podID="4b2cc236-34ca-4719-a938-2bf84d312d6e" containerID="b68e9367e957f21d68572c4f64f695fe0ce03480ea8247ed894464a60710b533" exitCode=0 Oct 30 07:25:14.688064 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.687890 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4" Oct 30 07:25:14.688064 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.687919 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4" event={"ID":"4b2cc236-34ca-4719-a938-2bf84d312d6e","Type":"ContainerDied","Data":"b68e9367e957f21d68572c4f64f695fe0ce03480ea8247ed894464a60710b533"} Oct 30 07:25:14.688064 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.687969 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4" event={"ID":"4b2cc236-34ca-4719-a938-2bf84d312d6e","Type":"ContainerDied","Data":"bb350ef0ba11009c2242267816deb7be39a474fe8a9046d7ea40a5b298d1c301"} Oct 30 07:25:14.688064 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.687996 4951 scope.go:122] "RemoveContainer" containerID="b68e9367e957f21d68572c4f64f695fe0ce03480ea8247ed894464a60710b533" Oct 30 07:25:14.690821 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.690799 4951 generic.go:356] "Generic (PLEG): container finished" podID="134d6e2c-a19a-49e2-9681-cceda422f787" containerID="e89b283b3c0335cb46a69d6a694d0bbf54c382ddf10ec1e45803cbaf6d1932c2" exitCode=0 Oct 30 07:25:14.691267 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.691244 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-7fd444d44c-4m5wn" event={"ID":"134d6e2c-a19a-49e2-9681-cceda422f787","Type":"ContainerDied","Data":"e89b283b3c0335cb46a69d6a694d0bbf54c382ddf10ec1e45803cbaf6d1932c2"} Oct 30 07:25:14.693666 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.693421 4951 generic.go:356] "Generic (PLEG): container finished" podID="18eba7a1-a108-4536-8740-fedd18b1d479" containerID="288f738c4a6e3750be63373294672e4d7c3ce8af8dcd9d0f358bb9cc21a0bb79" exitCode=0 Oct 30 07:25:14.693666 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.693501 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-769f7549d6-4cvk2" event={"ID":"18eba7a1-a108-4536-8740-fedd18b1d479","Type":"ContainerDied","Data":"288f738c4a6e3750be63373294672e4d7c3ce8af8dcd9d0f358bb9cc21a0bb79"} Oct 30 07:25:14.704294 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.704275 4951 scope.go:122] "RemoveContainer" containerID="b68e9367e957f21d68572c4f64f695fe0ce03480ea8247ed894464a60710b533" Oct 30 07:25:14.704801 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:25:14.704740 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b68e9367e957f21d68572c4f64f695fe0ce03480ea8247ed894464a60710b533\": container with ID starting with b68e9367e957f21d68572c4f64f695fe0ce03480ea8247ed894464a60710b533 not found: ID does not exist" containerID="b68e9367e957f21d68572c4f64f695fe0ce03480ea8247ed894464a60710b533" Oct 30 07:25:14.704801 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.704774 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b68e9367e957f21d68572c4f64f695fe0ce03480ea8247ed894464a60710b533"} err="failed to get container status \"b68e9367e957f21d68572c4f64f695fe0ce03480ea8247ed894464a60710b533\": rpc error: code = NotFound desc = could not find container \"b68e9367e957f21d68572c4f64f695fe0ce03480ea8247ed894464a60710b533\": container with ID starting with b68e9367e957f21d68572c4f64f695fe0ce03480ea8247ed894464a60710b533 not found: ID does not exist" Oct 30 07:25:14.727621 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.727557 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-769f7549d6-4cvk2" Oct 30 07:25:14.729105 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.729013 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwqjm\" (UniqueName: \"kubernetes.io/projected/4b2cc236-34ca-4719-a938-2bf84d312d6e-kube-api-access-nwqjm\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:14.729105 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.729044 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b2cc236-34ca-4719-a938-2bf84d312d6e-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:14.729105 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.729068 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b2cc236-34ca-4719-a938-2bf84d312d6e-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:14.729105 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.729081 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b2cc236-34ca-4719-a938-2bf84d312d6e-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:14.744664 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.743256 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4"] Oct 30 07:25:14.749334 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.749313 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8f428-75b9b78d66-hjfn4"] Oct 30 07:25:14.795709 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.795683 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-7fd444d44c-4m5wn" Oct 30 07:25:14.877600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.853095 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/134d6e2c-a19a-49e2-9681-cceda422f787-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/134d6e2c-a19a-49e2-9681-cceda422f787-plugins-config-volume\") pod \"134d6e2c-a19a-49e2-9681-cceda422f787\" (UID: \"134d6e2c-a19a-49e2-9681-cceda422f787\") " Oct 30 07:25:14.877977 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.877939 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18eba7a1-a108-4536-8740-fedd18b1d479-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18eba7a1-a108-4536-8740-fedd18b1d479-plugins-config-volume\") pod \"18eba7a1-a108-4536-8740-fedd18b1d479\" (UID: \"18eba7a1-a108-4536-8740-fedd18b1d479\") " Oct 30 07:25:14.878298 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.878273 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/134d6e2c-a19a-49e2-9681-cceda422f787-kube-api-access-zngpm\" (UniqueName: \"kubernetes.io/projected/134d6e2c-a19a-49e2-9681-cceda422f787-kube-api-access-zngpm\") pod \"134d6e2c-a19a-49e2-9681-cceda422f787\" (UID: \"134d6e2c-a19a-49e2-9681-cceda422f787\") " Oct 30 07:25:14.878501 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.878487 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18eba7a1-a108-4536-8740-fedd18b1d479-kube-api-access-v89nb\" (UniqueName: \"kubernetes.io/projected/18eba7a1-a108-4536-8740-fedd18b1d479-kube-api-access-v89nb\") pod \"18eba7a1-a108-4536-8740-fedd18b1d479\" (UID: \"18eba7a1-a108-4536-8740-fedd18b1d479\") " Oct 30 07:25:14.911597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.855383 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/134d6e2c-a19a-49e2-9681-cceda422f787-plugins-config-volume" pod "134d6e2c-a19a-49e2-9681-cceda422f787" (UID: "134d6e2c-a19a-49e2-9681-cceda422f787"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:25:14.911597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.883825 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18eba7a1-a108-4536-8740-fedd18b1d479-plugins-config-volume" pod "18eba7a1-a108-4536-8740-fedd18b1d479" (UID: "18eba7a1-a108-4536-8740-fedd18b1d479"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:25:14.942539 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.942499 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/134d6e2c-a19a-49e2-9681-cceda422f787-kube-api-access-zngpm" pod "134d6e2c-a19a-49e2-9681-cceda422f787" (UID: "134d6e2c-a19a-49e2-9681-cceda422f787"). InnerVolumeSpecName "kube-api-access-zngpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:25:14.957333 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.957265 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18eba7a1-a108-4536-8740-fedd18b1d479-kube-api-access-v89nb" pod "18eba7a1-a108-4536-8740-fedd18b1d479" (UID: "18eba7a1-a108-4536-8740-fedd18b1d479"). InnerVolumeSpecName "kube-api-access-v89nb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:25:14.986104 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.985970 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/134d6e2c-a19a-49e2-9681-cceda422f787-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:14.986527 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.986338 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18eba7a1-a108-4536-8740-fedd18b1d479-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:14.986527 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.986373 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zngpm\" (UniqueName: \"kubernetes.io/projected/134d6e2c-a19a-49e2-9681-cceda422f787-kube-api-access-zngpm\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:14.986527 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:14.986391 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v89nb\" (UniqueName: \"kubernetes.io/projected/18eba7a1-a108-4536-8740-fedd18b1d479-kube-api-access-v89nb\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:15.701950 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:15.701856 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-769f7549d6-4cvk2" Oct 30 07:25:15.701950 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:15.701897 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-769f7549d6-4cvk2" event={"ID":"18eba7a1-a108-4536-8740-fedd18b1d479","Type":"ContainerDied","Data":"5f259574517d97e5452f674460bb22586e07868d6821b89d27763391a51320a6"} Oct 30 07:25:15.702169 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:15.701966 4951 scope.go:122] "RemoveContainer" containerID="288f738c4a6e3750be63373294672e4d7c3ce8af8dcd9d0f358bb9cc21a0bb79" Oct 30 07:25:15.704650 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:15.704615 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-7fd444d44c-4m5wn" event={"ID":"134d6e2c-a19a-49e2-9681-cceda422f787","Type":"ContainerDied","Data":"e34030075264d57ac76c5421d68992f5c1c2c843b8dfe99826adf7308fa0ac12"} Oct 30 07:25:15.704650 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:15.704638 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-7fd444d44c-4m5wn" Oct 30 07:25:15.714133 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:15.714047 4951 scope.go:122] "RemoveContainer" containerID="e89b283b3c0335cb46a69d6a694d0bbf54c382ddf10ec1e45803cbaf6d1932c2" Oct 30 07:25:15.754742 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:15.754718 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-7fd444d44c-4m5wn"] Oct 30 07:25:15.773096 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:15.773055 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b2cc236-34ca-4719-a938-2bf84d312d6e" path="/var/lib/kubelet/pods/4b2cc236-34ca-4719-a938-2bf84d312d6e/volumes" Oct 30 07:25:15.773435 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:15.773421 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-7fd444d44c-4m5wn"] Oct 30 07:25:15.792547 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:15.792524 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8f428-769f7549d6-4cvk2"] Oct 30 07:25:15.800356 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:15.800329 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8f428-769f7549d6-4cvk2"] Oct 30 07:25:16.363061 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:16.363011 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-1beb92-8546db7586-cm9b4" podUID="2c506890-59f9-48b1-9d0c-c8878ffa730a" containerName="payload-processing" containerID="cri-o://4aae1c9712fc15ad3f22d5ba6a59ff13304abeb036564e7f4384974a1b8f1d43" gracePeriod=25 Oct 30 07:25:16.432743 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:16.432697 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1beb92-fc79c45ff-gpklb" podUID="070d9aa8-7438-4866-83fc-ac9a7e3a0443" containerName="payload-pre-processing" containerID="cri-o://19972948abc9b89d90c99a792ad81d9759c78e5fbfe994b4565a6e57a28c9e08" gracePeriod=25 Oct 30 07:25:16.715563 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:16.715531 4951 generic.go:356] "Generic (PLEG): container finished" podID="2c506890-59f9-48b1-9d0c-c8878ffa730a" containerID="4aae1c9712fc15ad3f22d5ba6a59ff13304abeb036564e7f4384974a1b8f1d43" exitCode=0 Oct 30 07:25:16.715762 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:16.715685 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1beb92-8546db7586-cm9b4" event={"ID":"2c506890-59f9-48b1-9d0c-c8878ffa730a","Type":"ContainerDied","Data":"4aae1c9712fc15ad3f22d5ba6a59ff13304abeb036564e7f4384974a1b8f1d43"} Oct 30 07:25:16.715845 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:16.715772 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1beb92-8546db7586-cm9b4" event={"ID":"2c506890-59f9-48b1-9d0c-c8878ffa730a","Type":"ContainerDied","Data":"8eeddb0c7bbedb5a5d593bab88d7b923dadec424170b69047b323d23267bd67e"} Oct 30 07:25:16.715845 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:16.715798 4951 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8eeddb0c7bbedb5a5d593bab88d7b923dadec424170b69047b323d23267bd67e" Oct 30 07:25:16.718611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:16.717801 4951 generic.go:356] "Generic (PLEG): container finished" podID="070d9aa8-7438-4866-83fc-ac9a7e3a0443" containerID="19972948abc9b89d90c99a792ad81d9759c78e5fbfe994b4565a6e57a28c9e08" exitCode=0 Oct 30 07:25:16.718611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:16.717861 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1beb92-fc79c45ff-gpklb" event={"ID":"070d9aa8-7438-4866-83fc-ac9a7e3a0443","Type":"ContainerDied","Data":"19972948abc9b89d90c99a792ad81d9759c78e5fbfe994b4565a6e57a28c9e08"} Oct 30 07:25:16.726399 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:16.726376 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1beb92-8546db7586-cm9b4" Oct 30 07:25:16.763612 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:16.763584 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1beb92-fc79c45ff-gpklb" Oct 30 07:25:16.819903 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:16.819869 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/070d9aa8-7438-4866-83fc-ac9a7e3a0443-kube-api-access-xr9c6\" (UniqueName: \"kubernetes.io/projected/070d9aa8-7438-4866-83fc-ac9a7e3a0443-kube-api-access-xr9c6\") pod \"070d9aa8-7438-4866-83fc-ac9a7e3a0443\" (UID: \"070d9aa8-7438-4866-83fc-ac9a7e3a0443\") " Oct 30 07:25:16.820128 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:16.819979 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/070d9aa8-7438-4866-83fc-ac9a7e3a0443-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/070d9aa8-7438-4866-83fc-ac9a7e3a0443-plugins-config-volume\") pod \"070d9aa8-7438-4866-83fc-ac9a7e3a0443\" (UID: \"070d9aa8-7438-4866-83fc-ac9a7e3a0443\") " Oct 30 07:25:16.820220 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:16.820157 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c506890-59f9-48b1-9d0c-c8878ffa730a-kube-api-access-snqz6\" (UniqueName: \"kubernetes.io/projected/2c506890-59f9-48b1-9d0c-c8878ffa730a-kube-api-access-snqz6\") pod \"2c506890-59f9-48b1-9d0c-c8878ffa730a\" (UID: \"2c506890-59f9-48b1-9d0c-c8878ffa730a\") " Oct 30 07:25:16.820278 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:16.820252 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c506890-59f9-48b1-9d0c-c8878ffa730a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c506890-59f9-48b1-9d0c-c8878ffa730a-plugins-config-volume\") pod \"2c506890-59f9-48b1-9d0c-c8878ffa730a\" (UID: \"2c506890-59f9-48b1-9d0c-c8878ffa730a\") " Oct 30 07:25:16.821512 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:16.821111 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c506890-59f9-48b1-9d0c-c8878ffa730a-plugins-config-volume" pod "2c506890-59f9-48b1-9d0c-c8878ffa730a" (UID: "2c506890-59f9-48b1-9d0c-c8878ffa730a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:25:16.821512 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:16.821307 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c506890-59f9-48b1-9d0c-c8878ffa730a-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:16.821512 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:16.821115 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/070d9aa8-7438-4866-83fc-ac9a7e3a0443-plugins-config-volume" pod "070d9aa8-7438-4866-83fc-ac9a7e3a0443" (UID: "070d9aa8-7438-4866-83fc-ac9a7e3a0443"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:25:16.823047 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:16.823017 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/070d9aa8-7438-4866-83fc-ac9a7e3a0443-kube-api-access-xr9c6" pod "070d9aa8-7438-4866-83fc-ac9a7e3a0443" (UID: "070d9aa8-7438-4866-83fc-ac9a7e3a0443"). InnerVolumeSpecName "kube-api-access-xr9c6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:25:16.824799 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:16.824773 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c506890-59f9-48b1-9d0c-c8878ffa730a-kube-api-access-snqz6" pod "2c506890-59f9-48b1-9d0c-c8878ffa730a" (UID: "2c506890-59f9-48b1-9d0c-c8878ffa730a"). InnerVolumeSpecName "kube-api-access-snqz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:25:16.922755 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:16.922726 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr9c6\" (UniqueName: \"kubernetes.io/projected/070d9aa8-7438-4866-83fc-ac9a7e3a0443-kube-api-access-xr9c6\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:16.922755 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:16.922750 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/070d9aa8-7438-4866-83fc-ac9a7e3a0443-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:16.922755 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:16.922762 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snqz6\" (UniqueName: \"kubernetes.io/projected/2c506890-59f9-48b1-9d0c-c8878ffa730a-kube-api-access-snqz6\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:17.727713 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:17.727398 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1beb92-fc79c45ff-gpklb" Oct 30 07:25:17.727713 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:17.727413 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1beb92-8546db7586-cm9b4" Oct 30 07:25:17.727713 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:17.727424 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1beb92-fc79c45ff-gpklb" event={"ID":"070d9aa8-7438-4866-83fc-ac9a7e3a0443","Type":"ContainerDied","Data":"c5188740c528f6de0623c1c9bb097db822c10b6046f18fceca78ef83738b4c69"} Oct 30 07:25:17.727713 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:17.727517 4951 scope.go:122] "RemoveContainer" containerID="19972948abc9b89d90c99a792ad81d9759c78e5fbfe994b4565a6e57a28c9e08" Oct 30 07:25:17.772840 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:17.772805 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="134d6e2c-a19a-49e2-9681-cceda422f787" path="/var/lib/kubelet/pods/134d6e2c-a19a-49e2-9681-cceda422f787/volumes" Oct 30 07:25:17.773462 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:17.773442 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18eba7a1-a108-4536-8740-fedd18b1d479" path="/var/lib/kubelet/pods/18eba7a1-a108-4536-8740-fedd18b1d479/volumes" Oct 30 07:25:17.803145 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:17.803090 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1beb92-8546db7586-cm9b4"] Oct 30 07:25:17.816295 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:17.816269 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1beb92-8546db7586-cm9b4"] Oct 30 07:25:17.850319 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:17.850291 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1beb92-fc79c45ff-gpklb"] Oct 30 07:25:17.857688 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:17.857663 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1beb92-fc79c45ff-gpklb"] Oct 30 07:25:19.772216 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:19.772184 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="070d9aa8-7438-4866-83fc-ac9a7e3a0443" path="/var/lib/kubelet/pods/070d9aa8-7438-4866-83fc-ac9a7e3a0443/volumes" Oct 30 07:25:19.772793 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:19.772775 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c506890-59f9-48b1-9d0c-c8878ffa730a" path="/var/lib/kubelet/pods/2c506890-59f9-48b1-9d0c-c8878ffa730a/volumes" Oct 30 07:25:21.962163 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:21.962124 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l"] Oct 30 07:25:21.962728 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:21.962343 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" podUID="03884451-d62c-4eea-827f-6e0c9359cbec" containerName="istio-proxy" containerID="cri-o://f860e49acd45a39557899aea264c0ffebaeb49db89cca82926091656add070f1" gracePeriod=30 Oct 30 07:25:23.024542 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:25:23.024503 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:25:27.239620 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.239525 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:25:27.323155 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.323113 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-workload-socket\") pod \"03884451-d62c-4eea-827f-6e0c9359cbec\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " Oct 30 07:25:27.323329 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.323213 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03884451-d62c-4eea-827f-6e0c9359cbec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03884451-d62c-4eea-827f-6e0c9359cbec-istio-podinfo\") pod \"03884451-d62c-4eea-827f-6e0c9359cbec\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " Oct 30 07:25:27.323528 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.323500 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-workload-socket" pod "03884451-d62c-4eea-827f-6e0c9359cbec" (UID: "03884451-d62c-4eea-827f-6e0c9359cbec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:25:27.323808 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.323782 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-workload-certs\") pod \"03884451-d62c-4eea-827f-6e0c9359cbec\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " Oct 30 07:25:27.323897 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.323846 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-istio-envoy\") pod \"03884451-d62c-4eea-827f-6e0c9359cbec\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " Oct 30 07:25:27.323954 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.323908 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03884451-d62c-4eea-827f-6e0c9359cbec-istio-token\" (UniqueName: \"kubernetes.io/projected/03884451-d62c-4eea-827f-6e0c9359cbec-istio-token\") pod \"03884451-d62c-4eea-827f-6e0c9359cbec\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " Oct 30 07:25:27.323954 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.323936 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-istio-data\") pod \"03884451-d62c-4eea-827f-6e0c9359cbec\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " Oct 30 07:25:27.324065 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.324041 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-workload-certs" pod "03884451-d62c-4eea-827f-6e0c9359cbec" (UID: "03884451-d62c-4eea-827f-6e0c9359cbec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:25:27.324190 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.324082 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03884451-d62c-4eea-827f-6e0c9359cbec-kube-api-access-fhps9\" (UniqueName: \"kubernetes.io/projected/03884451-d62c-4eea-827f-6e0c9359cbec-kube-api-access-fhps9\") pod \"03884451-d62c-4eea-827f-6e0c9359cbec\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " Oct 30 07:25:27.324190 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.324115 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-credential-socket\") pod \"03884451-d62c-4eea-827f-6e0c9359cbec\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " Oct 30 07:25:27.324748 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.324335 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03884451-d62c-4eea-827f-6e0c9359cbec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03884451-d62c-4eea-827f-6e0c9359cbec-istiod-ca-cert\") pod \"03884451-d62c-4eea-827f-6e0c9359cbec\" (UID: \"03884451-d62c-4eea-827f-6e0c9359cbec\") " Oct 30 07:25:27.324748 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.324520 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-istio-data" pod "03884451-d62c-4eea-827f-6e0c9359cbec" (UID: "03884451-d62c-4eea-827f-6e0c9359cbec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:25:27.324913 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.324820 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-credential-socket" pod "03884451-d62c-4eea-827f-6e0c9359cbec" (UID: "03884451-d62c-4eea-827f-6e0c9359cbec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:25:27.324984 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.324946 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03884451-d62c-4eea-827f-6e0c9359cbec-istiod-ca-cert" pod "03884451-d62c-4eea-827f-6e0c9359cbec" (UID: "03884451-d62c-4eea-827f-6e0c9359cbec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:25:27.325114 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.325081 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:27.325192 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.325111 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:27.325192 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.325129 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:27.325192 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.325143 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:27.327617 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.326969 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-istio-envoy" pod "03884451-d62c-4eea-827f-6e0c9359cbec" (UID: "03884451-d62c-4eea-827f-6e0c9359cbec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:25:27.327617 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.327096 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03884451-d62c-4eea-827f-6e0c9359cbec-istio-podinfo" pod "03884451-d62c-4eea-827f-6e0c9359cbec" (UID: "03884451-d62c-4eea-827f-6e0c9359cbec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:25:27.327617 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.327296 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03884451-d62c-4eea-827f-6e0c9359cbec-istio-token" pod "03884451-d62c-4eea-827f-6e0c9359cbec" (UID: "03884451-d62c-4eea-827f-6e0c9359cbec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:25:27.327617 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.327320 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03884451-d62c-4eea-827f-6e0c9359cbec-kube-api-access-fhps9" pod "03884451-d62c-4eea-827f-6e0c9359cbec" (UID: "03884451-d62c-4eea-827f-6e0c9359cbec"). InnerVolumeSpecName "kube-api-access-fhps9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:25:27.426771 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.426730 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhps9\" (UniqueName: \"kubernetes.io/projected/03884451-d62c-4eea-827f-6e0c9359cbec-kube-api-access-fhps9\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:27.426771 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.426771 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03884451-d62c-4eea-827f-6e0c9359cbec-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:27.427028 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.426786 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03884451-d62c-4eea-827f-6e0c9359cbec-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:27.427028 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.426800 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03884451-d62c-4eea-827f-6e0c9359cbec-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:27.427028 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.426814 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03884451-d62c-4eea-827f-6e0c9359cbec-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:27.938224 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.938187 4951 generic.go:356] "Generic (PLEG): container finished" podID="03884451-d62c-4eea-827f-6e0c9359cbec" containerID="f860e49acd45a39557899aea264c0ffebaeb49db89cca82926091656add070f1" exitCode=0 Oct 30 07:25:27.938451 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.938276 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" event={"ID":"03884451-d62c-4eea-827f-6e0c9359cbec","Type":"ContainerDied","Data":"f860e49acd45a39557899aea264c0ffebaeb49db89cca82926091656add070f1"} Oct 30 07:25:27.938451 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.938351 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" event={"ID":"03884451-d62c-4eea-827f-6e0c9359cbec","Type":"ContainerDied","Data":"c84a6c49df830564def2ca43b8e4bae666c8659b1b8be13430a43279500aeee1"} Oct 30 07:25:27.938451 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.938388 4951 scope.go:122] "RemoveContainer" containerID="f860e49acd45a39557899aea264c0ffebaeb49db89cca82926091656add070f1" Oct 30 07:25:27.938716 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.938605 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l" Oct 30 07:25:27.952267 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.952241 4951 scope.go:122] "RemoveContainer" containerID="9c409058287aed26d6bcb72f9661a2627c4114bc51e907727cba31aca4eab50c" Oct 30 07:25:27.967212 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.967184 4951 scope.go:122] "RemoveContainer" containerID="f860e49acd45a39557899aea264c0ffebaeb49db89cca82926091656add070f1" Oct 30 07:25:27.967768 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:25:27.967721 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f860e49acd45a39557899aea264c0ffebaeb49db89cca82926091656add070f1\": container with ID starting with f860e49acd45a39557899aea264c0ffebaeb49db89cca82926091656add070f1 not found: ID does not exist" containerID="f860e49acd45a39557899aea264c0ffebaeb49db89cca82926091656add070f1" Oct 30 07:25:27.967902 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.967776 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f860e49acd45a39557899aea264c0ffebaeb49db89cca82926091656add070f1"} err="failed to get container status \"f860e49acd45a39557899aea264c0ffebaeb49db89cca82926091656add070f1\": rpc error: code = NotFound desc = could not find container \"f860e49acd45a39557899aea264c0ffebaeb49db89cca82926091656add070f1\": container with ID starting with f860e49acd45a39557899aea264c0ffebaeb49db89cca82926091656add070f1 not found: ID does not exist" Oct 30 07:25:27.967902 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.967800 4951 scope.go:122] "RemoveContainer" containerID="9c409058287aed26d6bcb72f9661a2627c4114bc51e907727cba31aca4eab50c" Oct 30 07:25:27.968173 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:25:27.968147 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c409058287aed26d6bcb72f9661a2627c4114bc51e907727cba31aca4eab50c\": container with ID starting with 9c409058287aed26d6bcb72f9661a2627c4114bc51e907727cba31aca4eab50c not found: ID does not exist" containerID="9c409058287aed26d6bcb72f9661a2627c4114bc51e907727cba31aca4eab50c" Oct 30 07:25:27.968272 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:27.968184 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c409058287aed26d6bcb72f9661a2627c4114bc51e907727cba31aca4eab50c"} err="failed to get container status \"9c409058287aed26d6bcb72f9661a2627c4114bc51e907727cba31aca4eab50c\": rpc error: code = NotFound desc = could not find container \"9c409058287aed26d6bcb72f9661a2627c4114bc51e907727cba31aca4eab50c\": container with ID starting with 9c409058287aed26d6bcb72f9661a2627c4114bc51e907727cba31aca4eab50c not found: ID does not exist" Oct 30 07:25:28.008113 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:28.008087 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l"] Oct 30 07:25:28.010969 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:28.010949 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a8f428-openshift-default-5b5456c899-x9f4l"] Oct 30 07:25:29.784563 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:29.784529 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03884451-d62c-4eea-827f-6e0c9359cbec" path="/var/lib/kubelet/pods/03884451-d62c-4eea-827f-6e0c9359cbec/volumes" Oct 30 07:25:29.854072 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:29.854036 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-59f7e4-c8f484f6c-k9f2h"] Oct 30 07:25:29.854637 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:29.854624 4951 cpu_manager.go:414] "containerMap: removing container" podUID="41a7e771-8212-4749-a905-4826e1814eff" containerName="payload-processing" Oct 30 07:25:29.854684 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:29.854639 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41a7e771-8212-4749-a905-4826e1814eff" containerName="payload-processing" Oct 30 07:25:29.854684 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:29.854649 4951 cpu_manager.go:414] "containerMap: removing container" podUID="070d9aa8-7438-4866-83fc-ac9a7e3a0443" containerName="payload-pre-processing" Oct 30 07:25:29.854684 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:29.854655 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="070d9aa8-7438-4866-83fc-ac9a7e3a0443" containerName="payload-pre-processing" Oct 30 07:25:29.854684 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:29.854664 4951 cpu_manager.go:414] "containerMap: removing container" podUID="03884451-d62c-4eea-827f-6e0c9359cbec" containerName="istio-proxy" Oct 30 07:25:29.854684 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:29.854669 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03884451-d62c-4eea-827f-6e0c9359cbec" containerName="istio-proxy" Oct 30 07:25:29.854684 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:29.854676 4951 cpu_manager.go:414] "containerMap: removing container" podUID="2c506890-59f9-48b1-9d0c-c8878ffa730a" containerName="payload-processing" Oct 30 07:25:29.854684 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:29.854681 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c506890-59f9-48b1-9d0c-c8878ffa730a" containerName="payload-processing" Oct 30 07:25:29.854888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:29.854691 4951 cpu_manager.go:414] "containerMap: removing container" podUID="03884451-d62c-4eea-827f-6e0c9359cbec" containerName="istio-proxy" Oct 30 07:25:29.854888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:29.854696 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03884451-d62c-4eea-827f-6e0c9359cbec" containerName="istio-proxy" Oct 30 07:25:29.854888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:29.854706 4951 cpu_manager.go:414] "containerMap: removing container" podUID="134d6e2c-a19a-49e2-9681-cceda422f787" containerName="payload-pre-processing" Oct 30 07:25:29.854888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:29.854711 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="134d6e2c-a19a-49e2-9681-cceda422f787" containerName="payload-pre-processing" Oct 30 07:25:29.854888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:29.854722 4951 cpu_manager.go:414] "containerMap: removing container" podUID="4b2cc236-34ca-4719-a938-2bf84d312d6e" containerName="maas-api" Oct 30 07:25:29.854888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:29.854727 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b2cc236-34ca-4719-a938-2bf84d312d6e" containerName="maas-api" Oct 30 07:25:29.854888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:29.854741 4951 cpu_manager.go:414] "containerMap: removing container" podUID="18eba7a1-a108-4536-8740-fedd18b1d479" containerName="payload-processing" Oct 30 07:25:29.854888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:29.854745 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18eba7a1-a108-4536-8740-fedd18b1d479" containerName="payload-processing" Oct 30 07:25:29.854888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:29.854759 4951 cpu_manager.go:414] "containerMap: removing container" podUID="e312e239-4207-479a-a33e-d7769edd58b5" containerName="maas-api" Oct 30 07:25:29.854888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:29.854764 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e312e239-4207-479a-a33e-d7769edd58b5" containerName="maas-api" Oct 30 07:25:29.854888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:29.854865 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="134d6e2c-a19a-49e2-9681-cceda422f787" containerName="payload-pre-processing" Oct 30 07:25:29.854888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:29.854875 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b2cc236-34ca-4719-a938-2bf84d312d6e" containerName="maas-api" Oct 30 07:25:29.854888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:29.854884 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="18eba7a1-a108-4536-8740-fedd18b1d479" containerName="payload-processing" Oct 30 07:25:29.854888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:29.854893 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="e312e239-4207-479a-a33e-d7769edd58b5" containerName="maas-api" Oct 30 07:25:29.855293 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:29.854905 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="41a7e771-8212-4749-a905-4826e1814eff" containerName="payload-processing" Oct 30 07:25:29.855293 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:29.854911 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="070d9aa8-7438-4866-83fc-ac9a7e3a0443" containerName="payload-pre-processing" Oct 30 07:25:29.855293 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:29.854917 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c506890-59f9-48b1-9d0c-c8878ffa730a" containerName="payload-processing" Oct 30 07:25:29.855293 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:29.854923 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="03884451-d62c-4eea-827f-6e0c9359cbec" containerName="istio-proxy" Oct 30 07:25:29.855293 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:29.854929 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="03884451-d62c-4eea-827f-6e0c9359cbec" containerName="istio-proxy" Oct 30 07:25:29.860007 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:29.859989 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-59f7e4-c8f484f6c-k9f2h" Oct 30 07:25:29.879753 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:29.879723 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-59f7e4-c8f484f6c-k9f2h"] Oct 30 07:25:29.888674 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:29.888647 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-59f7e4\"" Oct 30 07:25:29.888819 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:29.888717 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-59f7e4-dockercfg-b4qkr\"" Oct 30 07:25:30.016598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:30.006440 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zswdp\" (UniqueName: \"kubernetes.io/projected/83322d7f-2879-4260-bc3c-a28fbc7b9b08-kube-api-access-zswdp\") pod \"payload-pre-processing-e2e-praxis-a-59f7e4-c8f484f6c-k9f2h\" (UID: \"83322d7f-2879-4260-bc3c-a28fbc7b9b08\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-59f7e4-c8f484f6c-k9f2h" Oct 30 07:25:30.016598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:30.006698 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83322d7f-2879-4260-bc3c-a28fbc7b9b08-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-59f7e4-c8f484f6c-k9f2h\" (UID: \"83322d7f-2879-4260-bc3c-a28fbc7b9b08\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-59f7e4-c8f484f6c-k9f2h" Oct 30 07:25:30.113645 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:30.111044 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zswdp\" (UniqueName: \"kubernetes.io/projected/83322d7f-2879-4260-bc3c-a28fbc7b9b08-kube-api-access-zswdp\") pod \"payload-pre-processing-e2e-praxis-a-59f7e4-c8f484f6c-k9f2h\" (UID: \"83322d7f-2879-4260-bc3c-a28fbc7b9b08\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-59f7e4-c8f484f6c-k9f2h" Oct 30 07:25:30.113645 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:30.111232 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83322d7f-2879-4260-bc3c-a28fbc7b9b08-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-59f7e4-c8f484f6c-k9f2h\" (UID: \"83322d7f-2879-4260-bc3c-a28fbc7b9b08\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-59f7e4-c8f484f6c-k9f2h" Oct 30 07:25:30.183595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:30.182810 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83322d7f-2879-4260-bc3c-a28fbc7b9b08-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-59f7e4-c8f484f6c-k9f2h\" (UID: \"83322d7f-2879-4260-bc3c-a28fbc7b9b08\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-59f7e4-c8f484f6c-k9f2h" Oct 30 07:25:30.202608 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:30.190526 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zswdp\" (UniqueName: \"kubernetes.io/projected/83322d7f-2879-4260-bc3c-a28fbc7b9b08-kube-api-access-zswdp\") pod \"payload-pre-processing-e2e-praxis-a-59f7e4-c8f484f6c-k9f2h\" (UID: \"83322d7f-2879-4260-bc3c-a28fbc7b9b08\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-59f7e4-c8f484f6c-k9f2h" Oct 30 07:25:30.475191 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:30.475112 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-59f7e4-c8f484f6c-k9f2h" Oct 30 07:25:30.783642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:30.783610 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-59f7e4-c8f484f6c-k9f2h"] Oct 30 07:25:30.784668 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:25:30.784638 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83322d7f_2879_4260_bc3c_a28fbc7b9b08.slice/crio-b687f03cf32f4eda36f28fe38665d46aac97133f19fae46a499e50328ac9d577 WatchSource:0}: Error finding container b687f03cf32f4eda36f28fe38665d46aac97133f19fae46a499e50328ac9d577: Status 404 returned error can't find the container with id b687f03cf32f4eda36f28fe38665d46aac97133f19fae46a499e50328ac9d577 Oct 30 07:25:30.966356 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:30.966302 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-59f7e4-c8f484f6c-k9f2h" event={"ID":"83322d7f-2879-4260-bc3c-a28fbc7b9b08","Type":"ContainerStarted","Data":"1fa0b74a37564aaf50ca0583a831b02dc224594a60a260819b2306f8250def6f"} Oct 30 07:25:30.966531 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:30.966375 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-59f7e4-c8f484f6c-k9f2h" event={"ID":"83322d7f-2879-4260-bc3c-a28fbc7b9b08","Type":"ContainerStarted","Data":"b687f03cf32f4eda36f28fe38665d46aac97133f19fae46a499e50328ac9d577"} Oct 30 07:25:30.966531 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:30.966475 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-59f7e4-c8f484f6c-k9f2h" Oct 30 07:25:31.008322 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:31.008264 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-59f7e4-c8f484f6c-k9f2h" podStartSLOduration=2.008245964 podStartE2EDuration="2.008245964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:31.006132905 +0000 UTC m=+1737.349180870" watchObservedRunningTime="2026-10-30 07:25:31.008245964 +0000 UTC m=+1737.351293929" Oct 30 07:25:31.589840 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:31.589797 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx"] Oct 30 07:25:31.603592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:31.596326 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx" Oct 30 07:25:31.615266 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:31.615237 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-5bb165-serving-cert\"" Oct 30 07:25:31.620075 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:31.620050 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx"] Oct 30 07:25:31.654192 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:31.649377 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx\" (UID: \"c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx" Oct 30 07:25:31.654192 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:31.649459 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx\" (UID: \"c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx" Oct 30 07:25:31.654192 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:31.649503 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hstl\" (UniqueName: \"kubernetes.io/projected/c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e-kube-api-access-8hstl\") pod \"maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx\" (UID: \"c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx" Oct 30 07:25:31.654192 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:31.649550 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e-metrics-certs\") pod \"maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx\" (UID: \"c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx" Oct 30 07:25:31.750456 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:31.750417 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx\" (UID: \"c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx" Oct 30 07:25:31.750739 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:31.750717 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hstl\" (UniqueName: \"kubernetes.io/projected/c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e-kube-api-access-8hstl\") pod \"maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx\" (UID: \"c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx" Oct 30 07:25:31.750918 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:31.750895 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e-metrics-certs\") pod \"maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx\" (UID: \"c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx" Oct 30 07:25:31.751107 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:31.751091 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx\" (UID: \"c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx" Oct 30 07:25:31.751237 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:31.751211 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp"] Oct 30 07:25:31.751502 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:31.751455 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" podUID="d5e6118a-e82b-4c28-b716-8e6cbaa81474" containerName="istio-proxy" containerID="cri-o://d4b39ef3fcb982b9c6046669228c3ab624c29fa877d6b6c6c46f3ba5fdc126bb" gracePeriod=30 Oct 30 07:25:31.752401 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:31.751786 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx\" (UID: \"c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx" Oct 30 07:25:31.754500 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:31.754476 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx\" (UID: \"c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx" Oct 30 07:25:31.756018 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:31.755997 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e-metrics-certs\") pod \"maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx\" (UID: \"c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx" Oct 30 07:25:31.776909 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:31.776887 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hstl\" (UniqueName: \"kubernetes.io/projected/c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e-kube-api-access-8hstl\") pod \"maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx\" (UID: \"c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx" Oct 30 07:25:31.922642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:31.922492 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx" Oct 30 07:25:32.298315 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:32.298280 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx"] Oct 30 07:25:32.298713 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:25:32.298689 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7be83ae_eff0_4c63_8e2c_9c3748ea7c2e.slice/crio-6b9bfea4ba789ebade04db8b880cdc9dc2cee4aabe0ce8b60b8bd7851c17c10e WatchSource:0}: Error finding container 6b9bfea4ba789ebade04db8b880cdc9dc2cee4aabe0ce8b60b8bd7851c17c10e: Status 404 returned error can't find the container with id 6b9bfea4ba789ebade04db8b880cdc9dc2cee4aabe0ce8b60b8bd7851c17c10e Oct 30 07:25:32.300322 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:32.300305 4951 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 07:25:33.019647 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:33.019604 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx" event={"ID":"c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e","Type":"ContainerStarted","Data":"712c8d76e7df07db56315e3a0b8bc9e9dbfe62e87e50e8d6cba13c603bd13b1f"} Oct 30 07:25:33.020096 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:33.019695 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx" event={"ID":"c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e","Type":"ContainerStarted","Data":"6b9bfea4ba789ebade04db8b880cdc9dc2cee4aabe0ce8b60b8bd7851c17c10e"} Oct 30 07:25:33.020096 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:33.019752 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx" Oct 30 07:25:33.066134 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:33.066080 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx" podStartSLOduration=1.6646823039999998 podStartE2EDuration="2.06606661s" totalImagesPullingTime="401.384306ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:33.065358317 +0000 UTC m=+1739.408406283" watchObservedRunningTime="2026-10-30 07:25:33.06606661 +0000 UTC m=+1739.409114595" Oct 30 07:25:33.332612 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:33.332495 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-17bd11-8c45f755d-2tr2x_a133e6d0-be93-4023-9831-7dacca337709/payload-processing/0.log" Oct 30 07:25:33.588311 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:33.588234 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-pbml8_861854e2-21ba-40a6-b1b5-e3e36bd0858f/payload-processing/0.log" Oct 30 07:25:33.860125 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:33.860044 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-pbml8_861854e2-21ba-40a6-b1b5-e3e36bd0858f/payload-processing/0.log" Oct 30 07:25:34.054704 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:34.054675 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-597d57-7889966d58-vb8fv_4152dc51-db44-4f78-b551-3310cf6137c2/payload-processing/0.log" Oct 30 07:25:34.263833 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:34.263791 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-597d57-7889966d58-vb8fv_4152dc51-db44-4f78-b551-3310cf6137c2/payload-processing/0.log" Oct 30 07:25:37.040006 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.039945 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:25:37.048780 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.048755 4951 generic.go:356] "Generic (PLEG): container finished" podID="d5e6118a-e82b-4c28-b716-8e6cbaa81474" containerID="d4b39ef3fcb982b9c6046669228c3ab624c29fa877d6b6c6c46f3ba5fdc126bb" exitCode=0 Oct 30 07:25:37.048916 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.048813 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" Oct 30 07:25:37.048916 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.048839 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" event={"ID":"d5e6118a-e82b-4c28-b716-8e6cbaa81474","Type":"ContainerDied","Data":"d4b39ef3fcb982b9c6046669228c3ab624c29fa877d6b6c6c46f3ba5fdc126bb"} Oct 30 07:25:37.049007 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.048952 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp" event={"ID":"d5e6118a-e82b-4c28-b716-8e6cbaa81474","Type":"ContainerDied","Data":"7f7e39f9669494a694d2d99c07e674500610f069b65c82c21e7a6c4617f14be5"} Oct 30 07:25:37.049007 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.048982 4951 scope.go:122] "RemoveContainer" containerID="d4b39ef3fcb982b9c6046669228c3ab624c29fa877d6b6c6c46f3ba5fdc126bb" Oct 30 07:25:37.059735 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.059701 4951 scope.go:122] "RemoveContainer" containerID="37f97d4e834152adcfa2b17a2477915d3bd49fafd25c081b312b763b8cd8905b" Oct 30 07:25:37.068847 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.068830 4951 scope.go:122] "RemoveContainer" containerID="d4b39ef3fcb982b9c6046669228c3ab624c29fa877d6b6c6c46f3ba5fdc126bb" Oct 30 07:25:37.069103 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:25:37.069085 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4b39ef3fcb982b9c6046669228c3ab624c29fa877d6b6c6c46f3ba5fdc126bb\": container with ID starting with d4b39ef3fcb982b9c6046669228c3ab624c29fa877d6b6c6c46f3ba5fdc126bb not found: ID does not exist" containerID="d4b39ef3fcb982b9c6046669228c3ab624c29fa877d6b6c6c46f3ba5fdc126bb" Oct 30 07:25:37.069166 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.069115 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4b39ef3fcb982b9c6046669228c3ab624c29fa877d6b6c6c46f3ba5fdc126bb"} err="failed to get container status \"d4b39ef3fcb982b9c6046669228c3ab624c29fa877d6b6c6c46f3ba5fdc126bb\": rpc error: code = NotFound desc = could not find container \"d4b39ef3fcb982b9c6046669228c3ab624c29fa877d6b6c6c46f3ba5fdc126bb\": container with ID starting with d4b39ef3fcb982b9c6046669228c3ab624c29fa877d6b6c6c46f3ba5fdc126bb not found: ID does not exist" Oct 30 07:25:37.069166 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.069132 4951 scope.go:122] "RemoveContainer" containerID="37f97d4e834152adcfa2b17a2477915d3bd49fafd25c081b312b763b8cd8905b" Oct 30 07:25:37.069355 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:25:37.069341 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37f97d4e834152adcfa2b17a2477915d3bd49fafd25c081b312b763b8cd8905b\": container with ID starting with 37f97d4e834152adcfa2b17a2477915d3bd49fafd25c081b312b763b8cd8905b not found: ID does not exist" containerID="37f97d4e834152adcfa2b17a2477915d3bd49fafd25c081b312b763b8cd8905b" Oct 30 07:25:37.069396 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.069358 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37f97d4e834152adcfa2b17a2477915d3bd49fafd25c081b312b763b8cd8905b"} err="failed to get container status \"37f97d4e834152adcfa2b17a2477915d3bd49fafd25c081b312b763b8cd8905b\": rpc error: code = NotFound desc = could not find container \"37f97d4e834152adcfa2b17a2477915d3bd49fafd25c081b312b763b8cd8905b\": container with ID starting with 37f97d4e834152adcfa2b17a2477915d3bd49fafd25c081b312b763b8cd8905b not found: ID does not exist" Oct 30 07:25:37.094402 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.094379 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istio-token\" (UniqueName: \"kubernetes.io/projected/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istio-token\") pod \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " Oct 30 07:25:37.094539 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.094458 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istio-envoy\") pod \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " Oct 30 07:25:37.094617 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.094545 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5e6118a-e82b-4c28-b716-8e6cbaa81474-kube-api-access-r8xjf\" (UniqueName: \"kubernetes.io/projected/d5e6118a-e82b-4c28-b716-8e6cbaa81474-kube-api-access-r8xjf\") pod \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " Oct 30 07:25:37.094735 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.094702 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-workload-socket\") pod \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " Oct 30 07:25:37.094809 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.094754 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istio-data\") pod \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " Oct 30 07:25:37.094809 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.094783 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-credential-socket\") pod \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " Oct 30 07:25:37.094907 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.094814 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istiod-ca-cert\") pod \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " Oct 30 07:25:37.094907 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.094891 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-workload-certs\") pod \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " Oct 30 07:25:37.095007 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.094951 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istio-podinfo\") pod \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\" (UID: \"d5e6118a-e82b-4c28-b716-8e6cbaa81474\") " Oct 30 07:25:37.095058 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.095023 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-workload-socket" pod "d5e6118a-e82b-4c28-b716-8e6cbaa81474" (UID: "d5e6118a-e82b-4c28-b716-8e6cbaa81474"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:25:37.095883 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.095124 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-credential-socket" pod "d5e6118a-e82b-4c28-b716-8e6cbaa81474" (UID: "d5e6118a-e82b-4c28-b716-8e6cbaa81474"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:25:37.095883 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.095276 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istio-data" pod "d5e6118a-e82b-4c28-b716-8e6cbaa81474" (UID: "d5e6118a-e82b-4c28-b716-8e6cbaa81474"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:25:37.095883 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.095554 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:37.095883 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.095595 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:37.095883 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.095610 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:37.095883 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.095708 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istiod-ca-cert" pod "d5e6118a-e82b-4c28-b716-8e6cbaa81474" (UID: "d5e6118a-e82b-4c28-b716-8e6cbaa81474"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:25:37.095883 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.095836 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-workload-certs" pod "d5e6118a-e82b-4c28-b716-8e6cbaa81474" (UID: "d5e6118a-e82b-4c28-b716-8e6cbaa81474"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:25:37.097278 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.097195 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istio-token" pod "d5e6118a-e82b-4c28-b716-8e6cbaa81474" (UID: "d5e6118a-e82b-4c28-b716-8e6cbaa81474"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:25:37.097475 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.097449 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istio-podinfo" pod "d5e6118a-e82b-4c28-b716-8e6cbaa81474" (UID: "d5e6118a-e82b-4c28-b716-8e6cbaa81474"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:25:37.097715 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.097455 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istio-envoy" pod "d5e6118a-e82b-4c28-b716-8e6cbaa81474" (UID: "d5e6118a-e82b-4c28-b716-8e6cbaa81474"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:25:37.097715 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.097709 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e6118a-e82b-4c28-b716-8e6cbaa81474-kube-api-access-r8xjf" pod "d5e6118a-e82b-4c28-b716-8e6cbaa81474" (UID: "d5e6118a-e82b-4c28-b716-8e6cbaa81474"). InnerVolumeSpecName "kube-api-access-r8xjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:25:37.197463 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.197428 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:37.197463 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.197459 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:37.197463 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.197468 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:37.197721 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.197477 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:37.197721 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.197486 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8xjf\" (UniqueName: \"kubernetes.io/projected/d5e6118a-e82b-4c28-b716-8e6cbaa81474-kube-api-access-r8xjf\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:37.197721 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.197495 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5e6118a-e82b-4c28-b716-8e6cbaa81474-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:37.448847 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.448805 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp"] Oct 30 07:25:37.452951 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.452913 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-96ca87-openshift-default-b9c858889-lgntp"] Oct 30 07:25:37.772624 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:37.772590 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5e6118a-e82b-4c28-b716-8e6cbaa81474" path="/var/lib/kubelet/pods/d5e6118a-e82b-4c28-b716-8e6cbaa81474/volumes" Oct 30 07:25:39.032623 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:39.032595 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx" Oct 30 07:25:42.010691 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:42.010659 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-59f7e4-c8f484f6c-k9f2h" Oct 30 07:25:47.944807 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:47.944716 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3a5866-546b46c8f5-89tgg"] Oct 30 07:25:52.948724 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:52.948631 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a5866-546b46c8f5-89tgg" podUID="0724f769-9a02-479b-b4f0-de7f37a72e99" containerName="payload-pre-processing" containerID="cri-o://3730bfd3761ed7b788b27c1ac17aba5a4227467508fba773b5de6008116c1c18" gracePeriod=25 Oct 30 07:25:53.053086 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:25:53.053034 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:25:53.219469 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:53.219129 4951 generic.go:356] "Generic (PLEG): container finished" podID="0724f769-9a02-479b-b4f0-de7f37a72e99" containerID="3730bfd3761ed7b788b27c1ac17aba5a4227467508fba773b5de6008116c1c18" exitCode=0 Oct 30 07:25:53.219469 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:53.219312 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a5866-546b46c8f5-89tgg" event={"ID":"0724f769-9a02-479b-b4f0-de7f37a72e99","Type":"ContainerDied","Data":"3730bfd3761ed7b788b27c1ac17aba5a4227467508fba773b5de6008116c1c18"} Oct 30 07:25:53.289629 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:53.289604 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a5866-546b46c8f5-89tgg" Oct 30 07:25:53.414964 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:53.414921 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0724f769-9a02-479b-b4f0-de7f37a72e99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0724f769-9a02-479b-b4f0-de7f37a72e99-plugins-config-volume\") pod \"0724f769-9a02-479b-b4f0-de7f37a72e99\" (UID: \"0724f769-9a02-479b-b4f0-de7f37a72e99\") " Oct 30 07:25:53.415147 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:53.415093 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0724f769-9a02-479b-b4f0-de7f37a72e99-kube-api-access-rwjl4\" (UniqueName: \"kubernetes.io/projected/0724f769-9a02-479b-b4f0-de7f37a72e99-kube-api-access-rwjl4\") pod \"0724f769-9a02-479b-b4f0-de7f37a72e99\" (UID: \"0724f769-9a02-479b-b4f0-de7f37a72e99\") " Oct 30 07:25:53.415411 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:53.415385 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0724f769-9a02-479b-b4f0-de7f37a72e99-plugins-config-volume" pod "0724f769-9a02-479b-b4f0-de7f37a72e99" (UID: "0724f769-9a02-479b-b4f0-de7f37a72e99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:25:53.415645 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:53.415630 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0724f769-9a02-479b-b4f0-de7f37a72e99-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:53.417227 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:53.417203 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0724f769-9a02-479b-b4f0-de7f37a72e99-kube-api-access-rwjl4" pod "0724f769-9a02-479b-b4f0-de7f37a72e99" (UID: "0724f769-9a02-479b-b4f0-de7f37a72e99"). InnerVolumeSpecName "kube-api-access-rwjl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:25:53.516802 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:53.516766 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwjl4\" (UniqueName: \"kubernetes.io/projected/0724f769-9a02-479b-b4f0-de7f37a72e99-kube-api-access-rwjl4\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:54.243644 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:54.241754 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a5866-546b46c8f5-89tgg" event={"ID":"0724f769-9a02-479b-b4f0-de7f37a72e99","Type":"ContainerDied","Data":"9ff79e1a3ee07c1d021793accc30f7c718ab414b91c7b37f2ef1048b953da330"} Oct 30 07:25:54.243644 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:54.241830 4951 scope.go:122] "RemoveContainer" containerID="3730bfd3761ed7b788b27c1ac17aba5a4227467508fba773b5de6008116c1c18" Oct 30 07:25:54.243644 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:54.241997 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3a5866-546b46c8f5-89tgg" Oct 30 07:25:54.295747 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:54.295712 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3a5866-546b46c8f5-89tgg"] Oct 30 07:25:54.307856 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:54.307824 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3a5866-546b46c8f5-89tgg"] Oct 30 07:25:55.350061 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:55.350022 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-210755-6b8cfcc74-jkcp6"] Oct 30 07:25:55.364372 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:55.364333 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-210755-6fc746bbc9-fw9hv"] Oct 30 07:25:55.856481 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:55.856450 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0724f769-9a02-479b-b4f0-de7f37a72e99" path="/var/lib/kubelet/pods/0724f769-9a02-479b-b4f0-de7f37a72e99/volumes" Oct 30 07:25:58.757472 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:58.757392 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2"] Oct 30 07:25:58.758730 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:58.758687 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2" podUID="0c27d255-21e3-475c-b23a-be2a023eafa8" containerName="maas-api" containerID="cri-o://4b2c10ecfbbce6e1b81e19574cff90cc2223b6d51e717f11926206861bdca675" gracePeriod=30 Oct 30 07:25:59.210665 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:59.199760 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2" Oct 30 07:25:59.282515 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:59.265948 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c27d255-21e3-475c-b23a-be2a023eafa8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c27d255-21e3-475c-b23a-be2a023eafa8-metrics-certs\") pod \"0c27d255-21e3-475c-b23a-be2a023eafa8\" (UID: \"0c27d255-21e3-475c-b23a-be2a023eafa8\") " Oct 30 07:25:59.282905 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:59.282869 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c27d255-21e3-475c-b23a-be2a023eafa8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c27d255-21e3-475c-b23a-be2a023eafa8-openshift-service-ca\") pod \"0c27d255-21e3-475c-b23a-be2a023eafa8\" (UID: \"0c27d255-21e3-475c-b23a-be2a023eafa8\") " Oct 30 07:25:59.293820 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:59.289727 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c27d255-21e3-475c-b23a-be2a023eafa8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c27d255-21e3-475c-b23a-be2a023eafa8-maas-api-tls\") pod \"0c27d255-21e3-475c-b23a-be2a023eafa8\" (UID: \"0c27d255-21e3-475c-b23a-be2a023eafa8\") " Oct 30 07:25:59.294260 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:59.291727 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c27d255-21e3-475c-b23a-be2a023eafa8-metrics-certs" pod "0c27d255-21e3-475c-b23a-be2a023eafa8" (UID: "0c27d255-21e3-475c-b23a-be2a023eafa8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:25:59.296080 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:59.294484 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c27d255-21e3-475c-b23a-be2a023eafa8-openshift-service-ca" pod "0c27d255-21e3-475c-b23a-be2a023eafa8" (UID: "0c27d255-21e3-475c-b23a-be2a023eafa8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:25:59.296360 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:59.296095 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c27d255-21e3-475c-b23a-be2a023eafa8-kube-api-access-wqx9c\" (UniqueName: \"kubernetes.io/projected/0c27d255-21e3-475c-b23a-be2a023eafa8-kube-api-access-wqx9c\") pod \"0c27d255-21e3-475c-b23a-be2a023eafa8\" (UID: \"0c27d255-21e3-475c-b23a-be2a023eafa8\") " Oct 30 07:25:59.296893 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:59.296813 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c27d255-21e3-475c-b23a-be2a023eafa8-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:59.296893 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:59.296839 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c27d255-21e3-475c-b23a-be2a023eafa8-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:59.299702 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:59.299662 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c27d255-21e3-475c-b23a-be2a023eafa8-kube-api-access-wqx9c" pod "0c27d255-21e3-475c-b23a-be2a023eafa8" (UID: "0c27d255-21e3-475c-b23a-be2a023eafa8"). InnerVolumeSpecName "kube-api-access-wqx9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:25:59.313561 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:59.306265 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c27d255-21e3-475c-b23a-be2a023eafa8-maas-api-tls" pod "0c27d255-21e3-475c-b23a-be2a023eafa8" (UID: "0c27d255-21e3-475c-b23a-be2a023eafa8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:25:59.323507 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:59.323404 4951 generic.go:356] "Generic (PLEG): container finished" podID="0c27d255-21e3-475c-b23a-be2a023eafa8" containerID="4b2c10ecfbbce6e1b81e19574cff90cc2223b6d51e717f11926206861bdca675" exitCode=0 Oct 30 07:25:59.323687 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:59.323518 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2" event={"ID":"0c27d255-21e3-475c-b23a-be2a023eafa8","Type":"ContainerDied","Data":"4b2c10ecfbbce6e1b81e19574cff90cc2223b6d51e717f11926206861bdca675"} Oct 30 07:25:59.323687 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:59.323555 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2" event={"ID":"0c27d255-21e3-475c-b23a-be2a023eafa8","Type":"ContainerDied","Data":"c607777744c3b963d60f42301c654a974f01ef09e5b69dd111977f84e552c6f4"} Oct 30 07:25:59.323687 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:59.323607 4951 scope.go:122] "RemoveContainer" containerID="4b2c10ecfbbce6e1b81e19574cff90cc2223b6d51e717f11926206861bdca675" Oct 30 07:25:59.323872 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:59.323721 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2" Oct 30 07:25:59.345755 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:59.345392 4951 scope.go:122] "RemoveContainer" containerID="4b2c10ecfbbce6e1b81e19574cff90cc2223b6d51e717f11926206861bdca675" Oct 30 07:25:59.345844 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:25:59.345771 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b2c10ecfbbce6e1b81e19574cff90cc2223b6d51e717f11926206861bdca675\": container with ID starting with 4b2c10ecfbbce6e1b81e19574cff90cc2223b6d51e717f11926206861bdca675 not found: ID does not exist" containerID="4b2c10ecfbbce6e1b81e19574cff90cc2223b6d51e717f11926206861bdca675" Oct 30 07:25:59.345844 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:59.345808 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b2c10ecfbbce6e1b81e19574cff90cc2223b6d51e717f11926206861bdca675"} err="failed to get container status \"4b2c10ecfbbce6e1b81e19574cff90cc2223b6d51e717f11926206861bdca675\": rpc error: code = NotFound desc = could not find container \"4b2c10ecfbbce6e1b81e19574cff90cc2223b6d51e717f11926206861bdca675\": container with ID starting with 4b2c10ecfbbce6e1b81e19574cff90cc2223b6d51e717f11926206861bdca675 not found: ID does not exist" Oct 30 07:25:59.381967 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:59.381866 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2"] Oct 30 07:25:59.384721 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:59.384689 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3a5866-78ff8c876-79kj2"] Oct 30 07:25:59.398172 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:59.398146 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c27d255-21e3-475c-b23a-be2a023eafa8-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:59.398172 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:59.398171 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqx9c\" (UniqueName: \"kubernetes.io/projected/0c27d255-21e3-475c-b23a-be2a023eafa8-kube-api-access-wqx9c\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:25:59.774920 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:25:59.774877 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c27d255-21e3-475c-b23a-be2a023eafa8" path="/var/lib/kubelet/pods/0c27d255-21e3-475c-b23a-be2a023eafa8/volumes" Oct 30 07:26:00.351344 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:00.351125 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-210755-6b8cfcc74-jkcp6" podUID="434bd254-777f-40f6-9141-a4a292dc8272" containerName="payload-processing" containerID="cri-o://750c1f23349c6d7c6cbf881354d67e6be0f29c8d2d958a4b1bc8b43077b45f71" gracePeriod=25 Oct 30 07:26:00.365445 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:00.365263 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-210755-6fc746bbc9-fw9hv" podUID="5ebb910d-c4e9-4680-bb51-cfdb40c30428" containerName="payload-pre-processing" containerID="cri-o://92124352f76bfaa3fa82ea3cd079897a4d3bfa7d1025d2c5a51302a072e4d48a" gracePeriod=25 Oct 30 07:26:00.416165 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:00.416131 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8"] Oct 30 07:26:00.416367 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:00.416345 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" podUID="4d9a6b57-897d-4c14-88d7-87190ce5c8b8" containerName="istio-proxy" containerID="cri-o://17e28c0df13c90588332a54e453eed4d487c1e2fd222074d79b2eadde214894e" gracePeriod=30 Oct 30 07:26:00.715012 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:00.714978 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-210755-6b8cfcc74-jkcp6" Oct 30 07:26:00.747561 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:00.747530 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-210755-6fc746bbc9-fw9hv" Oct 30 07:26:00.821047 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:00.821010 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ebb910d-c4e9-4680-bb51-cfdb40c30428-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ebb910d-c4e9-4680-bb51-cfdb40c30428-plugins-config-volume\") pod \"5ebb910d-c4e9-4680-bb51-cfdb40c30428\" (UID: \"5ebb910d-c4e9-4680-bb51-cfdb40c30428\") " Oct 30 07:26:00.821542 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:00.821157 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/434bd254-777f-40f6-9141-a4a292dc8272-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/434bd254-777f-40f6-9141-a4a292dc8272-plugins-config-volume\") pod \"434bd254-777f-40f6-9141-a4a292dc8272\" (UID: \"434bd254-777f-40f6-9141-a4a292dc8272\") " Oct 30 07:26:00.821542 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:00.821223 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/434bd254-777f-40f6-9141-a4a292dc8272-kube-api-access-nb4r9\" (UniqueName: \"kubernetes.io/projected/434bd254-777f-40f6-9141-a4a292dc8272-kube-api-access-nb4r9\") pod \"434bd254-777f-40f6-9141-a4a292dc8272\" (UID: \"434bd254-777f-40f6-9141-a4a292dc8272\") " Oct 30 07:26:00.821542 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:00.821355 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ebb910d-c4e9-4680-bb51-cfdb40c30428-kube-api-access-9sm5z\" (UniqueName: \"kubernetes.io/projected/5ebb910d-c4e9-4680-bb51-cfdb40c30428-kube-api-access-9sm5z\") pod \"5ebb910d-c4e9-4680-bb51-cfdb40c30428\" (UID: \"5ebb910d-c4e9-4680-bb51-cfdb40c30428\") " Oct 30 07:26:00.821542 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:00.821442 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ebb910d-c4e9-4680-bb51-cfdb40c30428-plugins-config-volume" pod "5ebb910d-c4e9-4680-bb51-cfdb40c30428" (UID: "5ebb910d-c4e9-4680-bb51-cfdb40c30428"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:26:00.822018 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:00.821888 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ebb910d-c4e9-4680-bb51-cfdb40c30428-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:00.822018 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:00.821897 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/434bd254-777f-40f6-9141-a4a292dc8272-plugins-config-volume" pod "434bd254-777f-40f6-9141-a4a292dc8272" (UID: "434bd254-777f-40f6-9141-a4a292dc8272"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:26:00.833046 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:00.833014 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/434bd254-777f-40f6-9141-a4a292dc8272-kube-api-access-nb4r9" pod "434bd254-777f-40f6-9141-a4a292dc8272" (UID: "434bd254-777f-40f6-9141-a4a292dc8272"). InnerVolumeSpecName "kube-api-access-nb4r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:26:00.835392 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:00.835365 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ebb910d-c4e9-4680-bb51-cfdb40c30428-kube-api-access-9sm5z" pod "5ebb910d-c4e9-4680-bb51-cfdb40c30428" (UID: "5ebb910d-c4e9-4680-bb51-cfdb40c30428"). InnerVolumeSpecName "kube-api-access-9sm5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:26:00.923987 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:00.923893 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sm5z\" (UniqueName: \"kubernetes.io/projected/5ebb910d-c4e9-4680-bb51-cfdb40c30428-kube-api-access-9sm5z\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:00.923987 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:00.923929 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/434bd254-777f-40f6-9141-a4a292dc8272-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:00.923987 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:00.923946 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb4r9\" (UniqueName: \"kubernetes.io/projected/434bd254-777f-40f6-9141-a4a292dc8272-kube-api-access-nb4r9\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:01.400263 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:01.391431 4951 generic.go:356] "Generic (PLEG): container finished" podID="434bd254-777f-40f6-9141-a4a292dc8272" containerID="750c1f23349c6d7c6cbf881354d67e6be0f29c8d2d958a4b1bc8b43077b45f71" exitCode=0 Oct 30 07:26:01.400263 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:01.391659 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-210755-6b8cfcc74-jkcp6" event={"ID":"434bd254-777f-40f6-9141-a4a292dc8272","Type":"ContainerDied","Data":"750c1f23349c6d7c6cbf881354d67e6be0f29c8d2d958a4b1bc8b43077b45f71"} Oct 30 07:26:01.400263 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:01.391739 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-210755-6b8cfcc74-jkcp6" event={"ID":"434bd254-777f-40f6-9141-a4a292dc8272","Type":"ContainerDied","Data":"f1b3cf8706dfcc61b06b72a942fcdf82b3743458c47d4b04f87f1c6aad41e5ab"} Oct 30 07:26:01.400263 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:01.391778 4951 scope.go:122] "RemoveContainer" containerID="750c1f23349c6d7c6cbf881354d67e6be0f29c8d2d958a4b1bc8b43077b45f71" Oct 30 07:26:01.400263 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:01.391900 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-210755-6b8cfcc74-jkcp6" Oct 30 07:26:01.420886 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:01.420843 4951 generic.go:356] "Generic (PLEG): container finished" podID="5ebb910d-c4e9-4680-bb51-cfdb40c30428" containerID="92124352f76bfaa3fa82ea3cd079897a4d3bfa7d1025d2c5a51302a072e4d48a" exitCode=0 Oct 30 07:26:01.421072 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:01.420922 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-210755-6fc746bbc9-fw9hv" event={"ID":"5ebb910d-c4e9-4680-bb51-cfdb40c30428","Type":"ContainerDied","Data":"92124352f76bfaa3fa82ea3cd079897a4d3bfa7d1025d2c5a51302a072e4d48a"} Oct 30 07:26:01.421072 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:01.421016 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-210755-6fc746bbc9-fw9hv" event={"ID":"5ebb910d-c4e9-4680-bb51-cfdb40c30428","Type":"ContainerDied","Data":"7f77e99b20bfb17e11f8eed3605338ec8eabb4ef35f3a308b327fc09646eff8c"} Oct 30 07:26:01.421159 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:01.421132 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-210755-6fc746bbc9-fw9hv" Oct 30 07:26:01.431286 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:01.431257 4951 scope.go:122] "RemoveContainer" containerID="750c1f23349c6d7c6cbf881354d67e6be0f29c8d2d958a4b1bc8b43077b45f71" Oct 30 07:26:01.438791 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:26:01.438746 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"750c1f23349c6d7c6cbf881354d67e6be0f29c8d2d958a4b1bc8b43077b45f71\": container with ID starting with 750c1f23349c6d7c6cbf881354d67e6be0f29c8d2d958a4b1bc8b43077b45f71 not found: ID does not exist" containerID="750c1f23349c6d7c6cbf881354d67e6be0f29c8d2d958a4b1bc8b43077b45f71" Oct 30 07:26:01.439032 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:01.438998 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"750c1f23349c6d7c6cbf881354d67e6be0f29c8d2d958a4b1bc8b43077b45f71"} err="failed to get container status \"750c1f23349c6d7c6cbf881354d67e6be0f29c8d2d958a4b1bc8b43077b45f71\": rpc error: code = NotFound desc = could not find container \"750c1f23349c6d7c6cbf881354d67e6be0f29c8d2d958a4b1bc8b43077b45f71\": container with ID starting with 750c1f23349c6d7c6cbf881354d67e6be0f29c8d2d958a4b1bc8b43077b45f71 not found: ID does not exist" Oct 30 07:26:01.439171 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:01.439154 4951 scope.go:122] "RemoveContainer" containerID="92124352f76bfaa3fa82ea3cd079897a4d3bfa7d1025d2c5a51302a072e4d48a" Oct 30 07:26:01.454819 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:01.454779 4951 scope.go:122] "RemoveContainer" containerID="92124352f76bfaa3fa82ea3cd079897a4d3bfa7d1025d2c5a51302a072e4d48a" Oct 30 07:26:01.455211 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:26:01.455179 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92124352f76bfaa3fa82ea3cd079897a4d3bfa7d1025d2c5a51302a072e4d48a\": container with ID starting with 92124352f76bfaa3fa82ea3cd079897a4d3bfa7d1025d2c5a51302a072e4d48a not found: ID does not exist" containerID="92124352f76bfaa3fa82ea3cd079897a4d3bfa7d1025d2c5a51302a072e4d48a" Oct 30 07:26:01.455322 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:01.455225 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92124352f76bfaa3fa82ea3cd079897a4d3bfa7d1025d2c5a51302a072e4d48a"} err="failed to get container status \"92124352f76bfaa3fa82ea3cd079897a4d3bfa7d1025d2c5a51302a072e4d48a\": rpc error: code = NotFound desc = could not find container \"92124352f76bfaa3fa82ea3cd079897a4d3bfa7d1025d2c5a51302a072e4d48a\": container with ID starting with 92124352f76bfaa3fa82ea3cd079897a4d3bfa7d1025d2c5a51302a072e4d48a not found: ID does not exist" Oct 30 07:26:01.456166 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:01.456137 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-210755-6b8cfcc74-jkcp6"] Oct 30 07:26:01.459216 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:01.459192 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-210755-6b8cfcc74-jkcp6"] Oct 30 07:26:01.492927 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:01.492892 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-210755-6fc746bbc9-fw9hv"] Oct 30 07:26:01.495877 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:01.495848 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-210755-6fc746bbc9-fw9hv"] Oct 30 07:26:01.773600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:01.773551 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="434bd254-777f-40f6-9141-a4a292dc8272" path="/var/lib/kubelet/pods/434bd254-777f-40f6-9141-a4a292dc8272/volumes" Oct 30 07:26:01.774026 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:01.774011 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ebb910d-c4e9-4680-bb51-cfdb40c30428" path="/var/lib/kubelet/pods/5ebb910d-c4e9-4680-bb51-cfdb40c30428/volumes" Oct 30 07:26:02.431909 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:02.431815 4951 generic.go:356] "Generic (PLEG): container finished" podID="4506abe9-6f43-4070-9b7b-64631f5be137" containerID="a1054e6f9ac21300a17a382d2aaf5d3039c4ef2d281d401b5440b7fc91f77dca" exitCode=0 Oct 30 07:26:02.431909 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:02.431865 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" event={"ID":"4506abe9-6f43-4070-9b7b-64631f5be137","Type":"ContainerDied","Data":"a1054e6f9ac21300a17a382d2aaf5d3039c4ef2d281d401b5440b7fc91f77dca"} Oct 30 07:26:02.432451 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:02.432337 4951 scope.go:122] "RemoveContainer" containerID="a1054e6f9ac21300a17a382d2aaf5d3039c4ef2d281d401b5440b7fc91f77dca" Oct 30 07:26:02.434259 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:02.434220 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:26:02.434367 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:02.434324 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:26:02.434405 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:02.434370 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:26:03.463041 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:03.463008 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" event={"ID":"4506abe9-6f43-4070-9b7b-64631f5be137","Type":"ContainerStarted","Data":"de7e9f9678a375bbdad9114fd00232caddcbc9c021d025f225dfc980640c68d3"} Oct 30 07:26:04.379197 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:04.378639 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:26:04.379952 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:04.379929 4951 patch_prober.go:29] interesting pod/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 07:26:04.380093 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:04.379966 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" podUID="4506abe9-6f43-4070-9b7b-64631f5be137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 07:26:05.379056 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:05.379021 4951 patch_prober.go:29] interesting pod/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 07:26:05.379474 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:05.379084 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" podUID="4506abe9-6f43-4070-9b7b-64631f5be137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 07:26:05.788279 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:05.788249 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:26:05.886148 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:05.886096 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-kube-api-access-n6fwv\" (UniqueName: \"kubernetes.io/projected/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-kube-api-access-n6fwv\") pod \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " Oct 30 07:26:05.886362 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:05.886172 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istio-token\" (UniqueName: \"kubernetes.io/projected/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istio-token\") pod \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " Oct 30 07:26:05.886362 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:05.886243 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istio-envoy\") pod \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " Oct 30 07:26:05.886362 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:05.886284 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istiod-ca-cert\") pod \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " Oct 30 07:26:05.886362 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:05.886330 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-workload-certs\") pod \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " Oct 30 07:26:05.886638 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:05.886425 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istio-podinfo\") pod \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " Oct 30 07:26:05.886638 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:05.886520 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-workload-socket\") pod \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " Oct 30 07:26:05.886638 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:05.886553 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-credential-socket\") pod \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " Oct 30 07:26:05.886638 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:05.886633 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istio-data\") pod \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\" (UID: \"4d9a6b57-897d-4c14-88d7-87190ce5c8b8\") " Oct 30 07:26:05.886904 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:05.886863 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-workload-certs" pod "4d9a6b57-897d-4c14-88d7-87190ce5c8b8" (UID: "4d9a6b57-897d-4c14-88d7-87190ce5c8b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:05.887245 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:05.887219 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:05.887659 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:05.887636 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-workload-socket" pod "4d9a6b57-897d-4c14-88d7-87190ce5c8b8" (UID: "4d9a6b57-897d-4c14-88d7-87190ce5c8b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:05.887967 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:05.887852 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istio-data" pod "4d9a6b57-897d-4c14-88d7-87190ce5c8b8" (UID: "4d9a6b57-897d-4c14-88d7-87190ce5c8b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:05.887967 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:05.887894 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istiod-ca-cert" pod "4d9a6b57-897d-4c14-88d7-87190ce5c8b8" (UID: "4d9a6b57-897d-4c14-88d7-87190ce5c8b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:26:05.889963 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:05.889926 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istio-envoy" pod "4d9a6b57-897d-4c14-88d7-87190ce5c8b8" (UID: "4d9a6b57-897d-4c14-88d7-87190ce5c8b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:05.890204 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:05.890105 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-credential-socket" pod "4d9a6b57-897d-4c14-88d7-87190ce5c8b8" (UID: "4d9a6b57-897d-4c14-88d7-87190ce5c8b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:05.890204 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:05.890161 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-kube-api-access-n6fwv" pod "4d9a6b57-897d-4c14-88d7-87190ce5c8b8" (UID: "4d9a6b57-897d-4c14-88d7-87190ce5c8b8"). InnerVolumeSpecName "kube-api-access-n6fwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:26:05.890794 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:05.890775 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istio-token" pod "4d9a6b57-897d-4c14-88d7-87190ce5c8b8" (UID: "4d9a6b57-897d-4c14-88d7-87190ce5c8b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:26:05.891314 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:05.891287 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istio-podinfo" pod "4d9a6b57-897d-4c14-88d7-87190ce5c8b8" (UID: "4d9a6b57-897d-4c14-88d7-87190ce5c8b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:26:05.988448 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:05.988397 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6fwv\" (UniqueName: \"kubernetes.io/projected/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-kube-api-access-n6fwv\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:05.988448 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:05.988434 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:05.988448 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:05.988443 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:05.988448 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:05.988453 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:05.988448 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:05.988461 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:05.988898 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:05.988469 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:05.988898 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:05.988478 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:05.988898 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:05.988486 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d9a6b57-897d-4c14-88d7-87190ce5c8b8-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:06.048645 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.047863 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm"] Oct 30 07:26:06.048812 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.048760 4951 cpu_manager.go:414] "containerMap: removing container" podUID="d5e6118a-e82b-4c28-b716-8e6cbaa81474" containerName="istio-proxy" Oct 30 07:26:06.048812 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.048781 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5e6118a-e82b-4c28-b716-8e6cbaa81474" containerName="istio-proxy" Oct 30 07:26:06.048960 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.048839 4951 cpu_manager.go:414] "containerMap: removing container" podUID="5ebb910d-c4e9-4680-bb51-cfdb40c30428" containerName="payload-pre-processing" Oct 30 07:26:06.048960 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.048847 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ebb910d-c4e9-4680-bb51-cfdb40c30428" containerName="payload-pre-processing" Oct 30 07:26:06.048960 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.048864 4951 cpu_manager.go:414] "containerMap: removing container" podUID="0c27d255-21e3-475c-b23a-be2a023eafa8" containerName="maas-api" Oct 30 07:26:06.048960 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.048870 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c27d255-21e3-475c-b23a-be2a023eafa8" containerName="maas-api" Oct 30 07:26:06.048960 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.048889 4951 cpu_manager.go:414] "containerMap: removing container" podUID="434bd254-777f-40f6-9141-a4a292dc8272" containerName="payload-processing" Oct 30 07:26:06.048960 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.048895 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="434bd254-777f-40f6-9141-a4a292dc8272" containerName="payload-processing" Oct 30 07:26:06.048960 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.048907 4951 cpu_manager.go:414] "containerMap: removing container" podUID="0724f769-9a02-479b-b4f0-de7f37a72e99" containerName="payload-pre-processing" Oct 30 07:26:06.048960 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.048914 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0724f769-9a02-479b-b4f0-de7f37a72e99" containerName="payload-pre-processing" Oct 30 07:26:06.048960 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.048930 4951 cpu_manager.go:414] "containerMap: removing container" podUID="4d9a6b57-897d-4c14-88d7-87190ce5c8b8" containerName="istio-proxy" Oct 30 07:26:06.048960 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.048937 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d9a6b57-897d-4c14-88d7-87190ce5c8b8" containerName="istio-proxy" Oct 30 07:26:06.049496 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.049077 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d9a6b57-897d-4c14-88d7-87190ce5c8b8" containerName="istio-proxy" Oct 30 07:26:06.049496 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.049093 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5e6118a-e82b-4c28-b716-8e6cbaa81474" containerName="istio-proxy" Oct 30 07:26:06.049496 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.049114 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5e6118a-e82b-4c28-b716-8e6cbaa81474" containerName="istio-proxy" Oct 30 07:26:06.049496 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.049126 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ebb910d-c4e9-4680-bb51-cfdb40c30428" containerName="payload-pre-processing" Oct 30 07:26:06.049496 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.049135 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c27d255-21e3-475c-b23a-be2a023eafa8" containerName="maas-api" Oct 30 07:26:06.049496 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.049146 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="434bd254-777f-40f6-9141-a4a292dc8272" containerName="payload-processing" Oct 30 07:26:06.049496 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.049156 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="0724f769-9a02-479b-b4f0-de7f37a72e99" containerName="payload-pre-processing" Oct 30 07:26:06.069158 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.069118 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm"] Oct 30 07:26:06.069353 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.069310 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" Oct 30 07:26:06.084993 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.084681 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-fbfb50\"/\"openshift-service-ca.crt\"" Oct 30 07:26:06.084993 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.084729 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-fbfb50\"/\"praxis-route-fbfb50-kserve-self-signed-certs\"" Oct 30 07:26:06.084993 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.084769 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-fbfb50\"/\"kube-root-ca.crt\"" Oct 30 07:26:06.084993 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.084810 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-fbfb50\"/\"default-dockercfg-8kpzc\"" Oct 30 07:26:06.089709 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.089667 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-dshm\") pod \"praxis-route-fbfb50-kserve-748968b77f-rhkkm\" (UID: \"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b\") " pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" Oct 30 07:26:06.089851 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.089730 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-model-cache\") pod \"praxis-route-fbfb50-kserve-748968b77f-rhkkm\" (UID: \"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b\") " pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" Oct 30 07:26:06.089851 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.089790 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-home\") pod \"praxis-route-fbfb50-kserve-748968b77f-rhkkm\" (UID: \"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b\") " pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" Oct 30 07:26:06.089851 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.089820 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-tmp-dir\") pod \"praxis-route-fbfb50-kserve-748968b77f-rhkkm\" (UID: \"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b\") " pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" Oct 30 07:26:06.090089 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.089854 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-tls-certs\") pod \"praxis-route-fbfb50-kserve-748968b77f-rhkkm\" (UID: \"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b\") " pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" Oct 30 07:26:06.090089 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.089944 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m8hd\" (UniqueName: \"kubernetes.io/projected/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-kube-api-access-4m8hd\") pod \"praxis-route-fbfb50-kserve-748968b77f-rhkkm\" (UID: \"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b\") " pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" Oct 30 07:26:06.191342 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.191301 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-dshm\") pod \"praxis-route-fbfb50-kserve-748968b77f-rhkkm\" (UID: \"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b\") " pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" Oct 30 07:26:06.191342 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.191346 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-model-cache\") pod \"praxis-route-fbfb50-kserve-748968b77f-rhkkm\" (UID: \"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b\") " pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" Oct 30 07:26:06.191645 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.191389 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-home\") pod \"praxis-route-fbfb50-kserve-748968b77f-rhkkm\" (UID: \"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b\") " pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" Oct 30 07:26:06.191645 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.191406 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-tmp-dir\") pod \"praxis-route-fbfb50-kserve-748968b77f-rhkkm\" (UID: \"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b\") " pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" Oct 30 07:26:06.191645 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.191427 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-tls-certs\") pod \"praxis-route-fbfb50-kserve-748968b77f-rhkkm\" (UID: \"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b\") " pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" Oct 30 07:26:06.191645 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.191517 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m8hd\" (UniqueName: \"kubernetes.io/projected/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-kube-api-access-4m8hd\") pod \"praxis-route-fbfb50-kserve-748968b77f-rhkkm\" (UID: \"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b\") " pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" Oct 30 07:26:06.192127 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.192098 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-home\") pod \"praxis-route-fbfb50-kserve-748968b77f-rhkkm\" (UID: \"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b\") " pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" Oct 30 07:26:06.192234 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.192146 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-model-cache\") pod \"praxis-route-fbfb50-kserve-748968b77f-rhkkm\" (UID: \"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b\") " pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" Oct 30 07:26:06.192293 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.192250 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-tmp-dir\") pod \"praxis-route-fbfb50-kserve-748968b77f-rhkkm\" (UID: \"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b\") " pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" Oct 30 07:26:06.193802 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.193773 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-dshm\") pod \"praxis-route-fbfb50-kserve-748968b77f-rhkkm\" (UID: \"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b\") " pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" Oct 30 07:26:06.194704 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.194686 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-tls-certs\") pod \"praxis-route-fbfb50-kserve-748968b77f-rhkkm\" (UID: \"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b\") " pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" Oct 30 07:26:06.217905 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.217871 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m8hd\" (UniqueName: \"kubernetes.io/projected/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-kube-api-access-4m8hd\") pod \"praxis-route-fbfb50-kserve-748968b77f-rhkkm\" (UID: \"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b\") " pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" Oct 30 07:26:06.389649 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.381964 4951 patch_prober.go:29] interesting pod/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 07:26:06.389649 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.382077 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" podUID="4506abe9-6f43-4070-9b7b-64631f5be137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 07:26:06.390216 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.389813 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" Oct 30 07:26:06.566598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.566053 4951 generic.go:356] "Generic (PLEG): container finished" podID="4d9a6b57-897d-4c14-88d7-87190ce5c8b8" containerID="17e28c0df13c90588332a54e453eed4d487c1e2fd222074d79b2eadde214894e" exitCode=0 Oct 30 07:26:06.566598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.566230 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" event={"ID":"4d9a6b57-897d-4c14-88d7-87190ce5c8b8","Type":"ContainerDied","Data":"17e28c0df13c90588332a54e453eed4d487c1e2fd222074d79b2eadde214894e"} Oct 30 07:26:06.566598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.566292 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" event={"ID":"4d9a6b57-897d-4c14-88d7-87190ce5c8b8","Type":"ContainerDied","Data":"1b9cc5f9920c6924bb3a194ceee9c6393961f6a5f67f1c3c966d6cadc3db197a"} Oct 30 07:26:06.566598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.566326 4951 scope.go:122] "RemoveContainer" containerID="17e28c0df13c90588332a54e453eed4d487c1e2fd222074d79b2eadde214894e" Oct 30 07:26:06.566985 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.566620 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8" Oct 30 07:26:06.618222 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.618146 4951 scope.go:122] "RemoveContainer" containerID="17e28c0df13c90588332a54e453eed4d487c1e2fd222074d79b2eadde214894e" Oct 30 07:26:06.618963 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:26:06.618888 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17e28c0df13c90588332a54e453eed4d487c1e2fd222074d79b2eadde214894e\": container with ID starting with 17e28c0df13c90588332a54e453eed4d487c1e2fd222074d79b2eadde214894e not found: ID does not exist" containerID="17e28c0df13c90588332a54e453eed4d487c1e2fd222074d79b2eadde214894e" Oct 30 07:26:06.618963 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.618928 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17e28c0df13c90588332a54e453eed4d487c1e2fd222074d79b2eadde214894e"} err="failed to get container status \"17e28c0df13c90588332a54e453eed4d487c1e2fd222074d79b2eadde214894e\": rpc error: code = NotFound desc = could not find container \"17e28c0df13c90588332a54e453eed4d487c1e2fd222074d79b2eadde214894e\": container with ID starting with 17e28c0df13c90588332a54e453eed4d487c1e2fd222074d79b2eadde214894e not found: ID does not exist" Oct 30 07:26:06.694091 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.694061 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8"] Oct 30 07:26:06.699019 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.698988 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-2dgt8"] Oct 30 07:26:06.984561 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:06.984532 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm"] Oct 30 07:26:06.988720 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:26:06.988093 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c4d60ab_c724_48a1_9be3_9f7ba9d9bc8b.slice/crio-4b748fd791e647be42c63b2921fcf80e729648c0c7b27024da9f2bbec1c7935d WatchSource:0}: Error finding container 4b748fd791e647be42c63b2921fcf80e729648c0c7b27024da9f2bbec1c7935d: Status 404 returned error can't find the container with id 4b748fd791e647be42c63b2921fcf80e729648c0c7b27024da9f2bbec1c7935d Oct 30 07:26:07.064602 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:07.050529 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh"] Oct 30 07:26:07.064602 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:07.051526 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" podUID="9b256cca-f09b-4a29-969d-da93436195ac" containerName="istio-proxy" containerID="cri-o://b777db902e8c5b70bfdba2a1ce2ed20d2d90e98d0bae47db958c1bc2101f69c3" gracePeriod=30 Oct 30 07:26:07.379295 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:07.379262 4951 patch_prober.go:29] interesting pod/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 07:26:07.379466 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:07.379315 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" podUID="4506abe9-6f43-4070-9b7b-64631f5be137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 07:26:07.546595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:07.545673 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:26:07.577348 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:07.577261 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" event={"ID":"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b","Type":"ContainerStarted","Data":"197a4e19929d6a1c2a6d8ac3edb7fc2f27865d1b710184561f2a7529311253a6"} Oct 30 07:26:07.577348 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:07.577321 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" event={"ID":"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b","Type":"ContainerStarted","Data":"4b748fd791e647be42c63b2921fcf80e729648c0c7b27024da9f2bbec1c7935d"} Oct 30 07:26:07.619923 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:07.619846 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" podStartSLOduration=1.61982785 podStartE2EDuration="1.61982785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:07.617750589 +0000 UTC m=+1773.960798589" watchObservedRunningTime="2026-10-30 07:26:07.61982785 +0000 UTC m=+1773.962875817" Oct 30 07:26:07.815739 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:07.815702 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d9a6b57-897d-4c14-88d7-87190ce5c8b8" path="/var/lib/kubelet/pods/4d9a6b57-897d-4c14-88d7-87190ce5c8b8/volumes" Oct 30 07:26:08.379465 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:08.379425 4951 patch_prober.go:29] interesting pod/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 07:26:08.379729 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:08.379476 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" podUID="4506abe9-6f43-4070-9b7b-64631f5be137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 07:26:08.390621 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:08.390564 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" Oct 30 07:26:09.384950 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:09.384915 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:26:09.386536 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:09.386508 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:26:12.337603 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.337512 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:26:12.436256 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.436217 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b256cca-f09b-4a29-969d-da93436195ac-kube-api-access-94j8f\" (UniqueName: \"kubernetes.io/projected/9b256cca-f09b-4a29-969d-da93436195ac-kube-api-access-94j8f\") pod \"9b256cca-f09b-4a29-969d-da93436195ac\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " Oct 30 07:26:12.436465 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.436327 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-credential-socket\") pod \"9b256cca-f09b-4a29-969d-da93436195ac\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " Oct 30 07:26:12.436465 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.436381 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b256cca-f09b-4a29-969d-da93436195ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b256cca-f09b-4a29-969d-da93436195ac-istiod-ca-cert\") pod \"9b256cca-f09b-4a29-969d-da93436195ac\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " Oct 30 07:26:12.436465 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.436429 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b256cca-f09b-4a29-969d-da93436195ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b256cca-f09b-4a29-969d-da93436195ac-istio-podinfo\") pod \"9b256cca-f09b-4a29-969d-da93436195ac\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " Oct 30 07:26:12.436683 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.436482 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-workload-certs\") pod \"9b256cca-f09b-4a29-969d-da93436195ac\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " Oct 30 07:26:12.436683 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.436559 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-istio-data\") pod \"9b256cca-f09b-4a29-969d-da93436195ac\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " Oct 30 07:26:12.436683 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.436633 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-istio-envoy\") pod \"9b256cca-f09b-4a29-969d-da93436195ac\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " Oct 30 07:26:12.436683 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.436667 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-workload-socket\") pod \"9b256cca-f09b-4a29-969d-da93436195ac\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " Oct 30 07:26:12.436898 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.436701 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b256cca-f09b-4a29-969d-da93436195ac-istio-token\" (UniqueName: \"kubernetes.io/projected/9b256cca-f09b-4a29-969d-da93436195ac-istio-token\") pod \"9b256cca-f09b-4a29-969d-da93436195ac\" (UID: \"9b256cca-f09b-4a29-969d-da93436195ac\") " Oct 30 07:26:12.436898 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.436707 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-credential-socket" pod "9b256cca-f09b-4a29-969d-da93436195ac" (UID: "9b256cca-f09b-4a29-969d-da93436195ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:12.437009 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.436903 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-istio-data" pod "9b256cca-f09b-4a29-969d-da93436195ac" (UID: "9b256cca-f09b-4a29-969d-da93436195ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:12.437230 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.437149 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-workload-certs" pod "9b256cca-f09b-4a29-969d-da93436195ac" (UID: "9b256cca-f09b-4a29-969d-da93436195ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:12.437230 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.437146 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-workload-socket" pod "9b256cca-f09b-4a29-969d-da93436195ac" (UID: "9b256cca-f09b-4a29-969d-da93436195ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:12.437752 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.437519 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b256cca-f09b-4a29-969d-da93436195ac-istiod-ca-cert" pod "9b256cca-f09b-4a29-969d-da93436195ac" (UID: "9b256cca-f09b-4a29-969d-da93436195ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:26:12.437849 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.437819 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:12.437849 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.437847 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:12.437989 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.437861 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:12.437989 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.437875 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:12.440291 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.440267 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b256cca-f09b-4a29-969d-da93436195ac-kube-api-access-94j8f" pod "9b256cca-f09b-4a29-969d-da93436195ac" (UID: "9b256cca-f09b-4a29-969d-da93436195ac"). InnerVolumeSpecName "kube-api-access-94j8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:26:12.440988 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.440956 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b256cca-f09b-4a29-969d-da93436195ac-istio-token" pod "9b256cca-f09b-4a29-969d-da93436195ac" (UID: "9b256cca-f09b-4a29-969d-da93436195ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:26:12.442084 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.441968 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-istio-envoy" pod "9b256cca-f09b-4a29-969d-da93436195ac" (UID: "9b256cca-f09b-4a29-969d-da93436195ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:12.442084 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.442056 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b256cca-f09b-4a29-969d-da93436195ac-istio-podinfo" pod "9b256cca-f09b-4a29-969d-da93436195ac" (UID: "9b256cca-f09b-4a29-969d-da93436195ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:26:12.538952 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.538919 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b256cca-f09b-4a29-969d-da93436195ac-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:12.538952 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.538947 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b256cca-f09b-4a29-969d-da93436195ac-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:12.538952 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.538955 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b256cca-f09b-4a29-969d-da93436195ac-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:12.539182 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.538963 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b256cca-f09b-4a29-969d-da93436195ac-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:12.539182 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.538973 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94j8f\" (UniqueName: \"kubernetes.io/projected/9b256cca-f09b-4a29-969d-da93436195ac-kube-api-access-94j8f\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:12.615357 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.615274 4951 generic.go:356] "Generic (PLEG): container finished" podID="9b256cca-f09b-4a29-969d-da93436195ac" containerID="b777db902e8c5b70bfdba2a1ce2ed20d2d90e98d0bae47db958c1bc2101f69c3" exitCode=0 Oct 30 07:26:12.615357 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.615329 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" Oct 30 07:26:12.615566 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.615362 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" event={"ID":"9b256cca-f09b-4a29-969d-da93436195ac","Type":"ContainerDied","Data":"b777db902e8c5b70bfdba2a1ce2ed20d2d90e98d0bae47db958c1bc2101f69c3"} Oct 30 07:26:12.615566 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.615409 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh" event={"ID":"9b256cca-f09b-4a29-969d-da93436195ac","Type":"ContainerDied","Data":"112ca14aed43fd3f0af671b008a9cd1c75c9cd6120bd08463468938fab8d407a"} Oct 30 07:26:12.615566 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.615433 4951 scope.go:122] "RemoveContainer" containerID="b777db902e8c5b70bfdba2a1ce2ed20d2d90e98d0bae47db958c1bc2101f69c3" Oct 30 07:26:12.626556 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.626535 4951 scope.go:122] "RemoveContainer" containerID="435e10662a06f6237fe5ca36b360304526f0065f595c5d55ebaf17f58b89612d" Oct 30 07:26:12.635672 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.635650 4951 scope.go:122] "RemoveContainer" containerID="b777db902e8c5b70bfdba2a1ce2ed20d2d90e98d0bae47db958c1bc2101f69c3" Oct 30 07:26:12.635950 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:26:12.635931 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b777db902e8c5b70bfdba2a1ce2ed20d2d90e98d0bae47db958c1bc2101f69c3\": container with ID starting with b777db902e8c5b70bfdba2a1ce2ed20d2d90e98d0bae47db958c1bc2101f69c3 not found: ID does not exist" containerID="b777db902e8c5b70bfdba2a1ce2ed20d2d90e98d0bae47db958c1bc2101f69c3" Oct 30 07:26:12.636015 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.635957 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b777db902e8c5b70bfdba2a1ce2ed20d2d90e98d0bae47db958c1bc2101f69c3"} err="failed to get container status \"b777db902e8c5b70bfdba2a1ce2ed20d2d90e98d0bae47db958c1bc2101f69c3\": rpc error: code = NotFound desc = could not find container \"b777db902e8c5b70bfdba2a1ce2ed20d2d90e98d0bae47db958c1bc2101f69c3\": container with ID starting with b777db902e8c5b70bfdba2a1ce2ed20d2d90e98d0bae47db958c1bc2101f69c3 not found: ID does not exist" Oct 30 07:26:12.636015 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.635972 4951 scope.go:122] "RemoveContainer" containerID="435e10662a06f6237fe5ca36b360304526f0065f595c5d55ebaf17f58b89612d" Oct 30 07:26:12.636180 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:26:12.636161 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"435e10662a06f6237fe5ca36b360304526f0065f595c5d55ebaf17f58b89612d\": container with ID starting with 435e10662a06f6237fe5ca36b360304526f0065f595c5d55ebaf17f58b89612d not found: ID does not exist" containerID="435e10662a06f6237fe5ca36b360304526f0065f595c5d55ebaf17f58b89612d" Oct 30 07:26:12.636240 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.636185 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"435e10662a06f6237fe5ca36b360304526f0065f595c5d55ebaf17f58b89612d"} err="failed to get container status \"435e10662a06f6237fe5ca36b360304526f0065f595c5d55ebaf17f58b89612d\": rpc error: code = NotFound desc = could not find container \"435e10662a06f6237fe5ca36b360304526f0065f595c5d55ebaf17f58b89612d\": container with ID starting with 435e10662a06f6237fe5ca36b360304526f0065f595c5d55ebaf17f58b89612d not found: ID does not exist" Oct 30 07:26:12.671382 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.671346 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh"] Oct 30 07:26:12.675292 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:12.675258 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-3a5866-openshift-default-6b98b4b7cc-r9fdh"] Oct 30 07:26:13.802594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:13.787313 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b256cca-f09b-4a29-969d-da93436195ac" path="/var/lib/kubelet/pods/9b256cca-f09b-4a29-969d-da93436195ac/volumes" Oct 30 07:26:15.494341 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.493313 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f"] Oct 30 07:26:15.495609 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.495563 4951 cpu_manager.go:414] "containerMap: removing container" podUID="d5e6118a-e82b-4c28-b716-8e6cbaa81474" containerName="istio-proxy" Oct 30 07:26:15.495609 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.495607 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5e6118a-e82b-4c28-b716-8e6cbaa81474" containerName="istio-proxy" Oct 30 07:26:15.495818 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.495630 4951 cpu_manager.go:414] "containerMap: removing container" podUID="9b256cca-f09b-4a29-969d-da93436195ac" containerName="istio-proxy" Oct 30 07:26:15.495818 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.495638 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b256cca-f09b-4a29-969d-da93436195ac" containerName="istio-proxy" Oct 30 07:26:15.495939 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.495856 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b256cca-f09b-4a29-969d-da93436195ac" containerName="istio-proxy" Oct 30 07:26:15.495939 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.495882 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b256cca-f09b-4a29-969d-da93436195ac" containerName="istio-proxy" Oct 30 07:26:15.501681 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.501656 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:26:15.518986 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.518965 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-f4a665-openshift-default-dockercfg-bk7hc\"" Oct 30 07:26:15.538460 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.538430 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f"] Oct 30 07:26:15.575007 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.574968 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-istio-envoy\") pod \"e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:26:15.575229 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.575051 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-workload-certs\") pod \"e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:26:15.575229 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.575158 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e446711-8b80-43b4-9c61-b69b3f8a320b-istiod-ca-cert\") pod \"e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:26:15.575229 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.575189 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ctd8\" (UniqueName: \"kubernetes.io/projected/0e446711-8b80-43b4-9c61-b69b3f8a320b-kube-api-access-4ctd8\") pod \"e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:26:15.575229 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.575215 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-workload-socket\") pod \"e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:26:15.575507 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.575241 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-istio-data\") pod \"e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:26:15.575507 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.575261 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e446711-8b80-43b4-9c61-b69b3f8a320b-istio-podinfo\") pod \"e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:26:15.575507 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.575320 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e446711-8b80-43b4-9c61-b69b3f8a320b-istio-token\") pod \"e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:26:15.575507 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.575338 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-credential-socket\") pod \"e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:26:15.687596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.678923 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e446711-8b80-43b4-9c61-b69b3f8a320b-istio-token\") pod \"e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:26:15.687596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.678981 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-credential-socket\") pod \"e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:26:15.687596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.679032 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-istio-envoy\") pod \"e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:26:15.687596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.679118 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-workload-certs\") pod \"e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:26:15.687596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.679234 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e446711-8b80-43b4-9c61-b69b3f8a320b-istiod-ca-cert\") pod \"e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:26:15.687596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.679270 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ctd8\" (UniqueName: \"kubernetes.io/projected/0e446711-8b80-43b4-9c61-b69b3f8a320b-kube-api-access-4ctd8\") pod \"e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:26:15.687596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.679310 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-workload-socket\") pod \"e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:26:15.687596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.679353 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-istio-data\") pod \"e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:26:15.687596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.679380 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e446711-8b80-43b4-9c61-b69b3f8a320b-istio-podinfo\") pod \"e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:26:15.694195 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.694154 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-workload-certs\") pod \"e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:26:15.695130 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.695101 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-credential-socket\") pod \"e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:26:15.706597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.696676 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e446711-8b80-43b4-9c61-b69b3f8a320b-istiod-ca-cert\") pod \"e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:26:15.733639 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.707214 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-workload-socket\") pod \"e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:26:15.733908 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.707436 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-istio-data\") pod \"e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:26:15.734047 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.730893 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-istio-envoy\") pod \"e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:26:15.756228 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.756193 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e446711-8b80-43b4-9c61-b69b3f8a320b-istio-podinfo\") pod \"e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:26:15.766049 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.756960 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ctd8\" (UniqueName: \"kubernetes.io/projected/0e446711-8b80-43b4-9c61-b69b3f8a320b-kube-api-access-4ctd8\") pod \"e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:26:15.766049 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.759205 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e446711-8b80-43b4-9c61-b69b3f8a320b-istio-token\") pod \"e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:26:15.822321 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:15.822050 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:26:16.183236 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:16.183206 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f"] Oct 30 07:26:16.184814 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:26:16.184786 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e446711_8b80_43b4_9c61_b69b3f8a320b.slice/crio-ae63c84f1f748a5e786daa8ba55cdefad6d32269a9bf9384e0c538c608c7b529 WatchSource:0}: Error finding container ae63c84f1f748a5e786daa8ba55cdefad6d32269a9bf9384e0c538c608c7b529: Status 404 returned error can't find the container with id ae63c84f1f748a5e786daa8ba55cdefad6d32269a9bf9384e0c538c608c7b529 Oct 30 07:26:16.187431 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:16.187390 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:26:16.187551 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:16.187471 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:26:16.187551 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:16.187505 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:26:16.390464 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:16.390428 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" Oct 30 07:26:16.403912 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:16.403880 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" Oct 30 07:26:16.652655 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:16.652618 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" event={"ID":"0e446711-8b80-43b4-9c61-b69b3f8a320b","Type":"ContainerStarted","Data":"3d9cdfe263aad4b6fc741769343485d1f64f2b77e754cd96db90e7467c0df063"} Oct 30 07:26:16.653192 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:16.652680 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" event={"ID":"0e446711-8b80-43b4-9c61-b69b3f8a320b","Type":"ContainerStarted","Data":"ae63c84f1f748a5e786daa8ba55cdefad6d32269a9bf9384e0c538c608c7b529"} Oct 30 07:26:16.665565 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:16.665540 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" Oct 30 07:26:16.697498 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:16.697447 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" podStartSLOduration=1.697434149 podStartE2EDuration="1.697434149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:16.697172403 +0000 UTC m=+1783.040220403" watchObservedRunningTime="2026-10-30 07:26:16.697434149 +0000 UTC m=+1783.040482180" Oct 30 07:26:16.822410 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:16.822373 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:26:17.829020 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:17.828992 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:26:18.689596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:18.682654 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:26:18.689596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:18.683910 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:26:21.159258 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:21.159221 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f4a665-c894f7bc5-ccjfc"] Oct 30 07:26:21.159907 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:21.159889 4951 cpu_manager.go:414] "containerMap: removing container" podUID="9b256cca-f09b-4a29-969d-da93436195ac" containerName="istio-proxy" Oct 30 07:26:21.159907 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:21.159906 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b256cca-f09b-4a29-969d-da93436195ac" containerName="istio-proxy" Oct 30 07:26:21.163882 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:21.163859 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4a665-c894f7bc5-ccjfc" Oct 30 07:26:21.186344 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:21.186317 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-f4a665-dockercfg-x24p9\"" Oct 30 07:26:21.189523 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:21.189495 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f4a665-c894f7bc5-ccjfc"] Oct 30 07:26:21.273392 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:21.273350 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw7jc\" (UniqueName: \"kubernetes.io/projected/9ab4c115-8064-41df-96e2-be5cde2e4846-kube-api-access-vw7jc\") pod \"payload-pre-processing-e2e-praxis-b-f4a665-c894f7bc5-ccjfc\" (UID: \"9ab4c115-8064-41df-96e2-be5cde2e4846\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4a665-c894f7bc5-ccjfc" Oct 30 07:26:21.273624 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:21.273432 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ab4c115-8064-41df-96e2-be5cde2e4846-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f4a665-c894f7bc5-ccjfc\" (UID: \"9ab4c115-8064-41df-96e2-be5cde2e4846\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4a665-c894f7bc5-ccjfc" Oct 30 07:26:21.279056 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:21.279028 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-f4a665\"" Oct 30 07:26:21.390646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:21.376241 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ab4c115-8064-41df-96e2-be5cde2e4846-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f4a665-c894f7bc5-ccjfc\" (UID: \"9ab4c115-8064-41df-96e2-be5cde2e4846\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4a665-c894f7bc5-ccjfc" Oct 30 07:26:21.391256 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:21.391215 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw7jc\" (UniqueName: \"kubernetes.io/projected/9ab4c115-8064-41df-96e2-be5cde2e4846-kube-api-access-vw7jc\") pod \"payload-pre-processing-e2e-praxis-b-f4a665-c894f7bc5-ccjfc\" (UID: \"9ab4c115-8064-41df-96e2-be5cde2e4846\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4a665-c894f7bc5-ccjfc" Oct 30 07:26:21.394978 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:21.394946 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ab4c115-8064-41df-96e2-be5cde2e4846-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f4a665-c894f7bc5-ccjfc\" (UID: \"9ab4c115-8064-41df-96e2-be5cde2e4846\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4a665-c894f7bc5-ccjfc" Oct 30 07:26:21.437559 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:21.437477 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw7jc\" (UniqueName: \"kubernetes.io/projected/9ab4c115-8064-41df-96e2-be5cde2e4846-kube-api-access-vw7jc\") pod \"payload-pre-processing-e2e-praxis-b-f4a665-c894f7bc5-ccjfc\" (UID: \"9ab4c115-8064-41df-96e2-be5cde2e4846\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4a665-c894f7bc5-ccjfc" Oct 30 07:26:21.479865 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:21.479811 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4a665-c894f7bc5-ccjfc" Oct 30 07:26:21.792744 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:21.792720 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f4a665-c894f7bc5-ccjfc"] Oct 30 07:26:21.793876 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:26:21.793849 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ab4c115_8064_41df_96e2_be5cde2e4846.slice/crio-01570cd99f96a5b02223dfd7d66c66358d83f502bba7c2b3417ea6f57db1bbbe WatchSource:0}: Error finding container 01570cd99f96a5b02223dfd7d66c66358d83f502bba7c2b3417ea6f57db1bbbe: Status 404 returned error can't find the container with id 01570cd99f96a5b02223dfd7d66c66358d83f502bba7c2b3417ea6f57db1bbbe Oct 30 07:26:22.644788 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:22.644750 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-17bd11-8c45f755d-2tr2x"] Oct 30 07:26:22.645269 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:22.644990 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-8c45f755d-2tr2x" podUID="a133e6d0-be93-4023-9831-7dacca337709" containerName="payload-processing" containerID="cri-o://7473c8c2a202f23e616dec718ac0d09fdd5983efaa5d706051ad8f72d9e0c97c" gracePeriod=30 Oct 30 07:26:22.683310 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:22.683277 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-58bf9bd7f9-7m5cm"] Oct 30 07:26:22.683512 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:22.683485 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-58bf9bd7f9-7m5cm" podUID="41e60656-b3e9-4438-b73e-91e2632e452f" containerName="payload-pre-processing" containerID="cri-o://44806be4f52b4ef13df780a24a249c52d8ab060c919c26ae359e10c2533aeb53" gracePeriod=30 Oct 30 07:26:22.714491 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:22.714454 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4a665-c894f7bc5-ccjfc" event={"ID":"9ab4c115-8064-41df-96e2-be5cde2e4846","Type":"ContainerStarted","Data":"e08306491db9bb8603107230bdf350736b06f5f59d686b6e3cf96728855069e8"} Oct 30 07:26:22.714663 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:22.714517 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4a665-c894f7bc5-ccjfc" event={"ID":"9ab4c115-8064-41df-96e2-be5cde2e4846","Type":"ContainerStarted","Data":"01570cd99f96a5b02223dfd7d66c66358d83f502bba7c2b3417ea6f57db1bbbe"} Oct 30 07:26:22.715274 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:22.715249 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4a665-c894f7bc5-ccjfc" Oct 30 07:26:22.799715 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:22.799661 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4a665-c894f7bc5-ccjfc" podStartSLOduration=1.799645672 podStartE2EDuration="1.799645672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:22.796304504 +0000 UTC m=+1789.139352682" watchObservedRunningTime="2026-10-30 07:26:22.799645672 +0000 UTC m=+1789.142693636" Oct 30 07:26:23.000915 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:23.000886 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-8c45f755d-2tr2x" Oct 30 07:26:23.026841 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:26:23.026802 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:26:23.060961 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:23.060309 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-58bf9bd7f9-7m5cm" Oct 30 07:26:23.114995 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:23.114962 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a133e6d0-be93-4023-9831-7dacca337709-kube-api-access-wnn9d\" (UniqueName: \"kubernetes.io/projected/a133e6d0-be93-4023-9831-7dacca337709-kube-api-access-wnn9d\") pod \"a133e6d0-be93-4023-9831-7dacca337709\" (UID: \"a133e6d0-be93-4023-9831-7dacca337709\") " Oct 30 07:26:23.114995 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:23.114997 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41e60656-b3e9-4438-b73e-91e2632e452f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e60656-b3e9-4438-b73e-91e2632e452f-plugins-config-volume\") pod \"41e60656-b3e9-4438-b73e-91e2632e452f\" (UID: \"41e60656-b3e9-4438-b73e-91e2632e452f\") " Oct 30 07:26:23.115281 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:23.115035 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41e60656-b3e9-4438-b73e-91e2632e452f-kube-api-access-qnl4d\" (UniqueName: \"kubernetes.io/projected/41e60656-b3e9-4438-b73e-91e2632e452f-kube-api-access-qnl4d\") pod \"41e60656-b3e9-4438-b73e-91e2632e452f\" (UID: \"41e60656-b3e9-4438-b73e-91e2632e452f\") " Oct 30 07:26:23.115281 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:23.115200 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a133e6d0-be93-4023-9831-7dacca337709-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a133e6d0-be93-4023-9831-7dacca337709-plugins-config-volume\") pod \"a133e6d0-be93-4023-9831-7dacca337709\" (UID: \"a133e6d0-be93-4023-9831-7dacca337709\") " Oct 30 07:26:23.115520 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:23.115497 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e60656-b3e9-4438-b73e-91e2632e452f-plugins-config-volume" pod "41e60656-b3e9-4438-b73e-91e2632e452f" (UID: "41e60656-b3e9-4438-b73e-91e2632e452f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:26:23.115670 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:23.115566 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a133e6d0-be93-4023-9831-7dacca337709-plugins-config-volume" pod "a133e6d0-be93-4023-9831-7dacca337709" (UID: "a133e6d0-be93-4023-9831-7dacca337709"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:26:23.115768 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:23.115753 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a133e6d0-be93-4023-9831-7dacca337709-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:23.115813 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:23.115771 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e60656-b3e9-4438-b73e-91e2632e452f-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:23.117234 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:23.117212 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e60656-b3e9-4438-b73e-91e2632e452f-kube-api-access-qnl4d" pod "41e60656-b3e9-4438-b73e-91e2632e452f" (UID: "41e60656-b3e9-4438-b73e-91e2632e452f"). InnerVolumeSpecName "kube-api-access-qnl4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:26:23.117287 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:23.117252 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a133e6d0-be93-4023-9831-7dacca337709-kube-api-access-wnn9d" pod "a133e6d0-be93-4023-9831-7dacca337709" (UID: "a133e6d0-be93-4023-9831-7dacca337709"). InnerVolumeSpecName "kube-api-access-wnn9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:26:23.217631 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:23.217527 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnn9d\" (UniqueName: \"kubernetes.io/projected/a133e6d0-be93-4023-9831-7dacca337709-kube-api-access-wnn9d\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:23.217631 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:23.217554 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnl4d\" (UniqueName: \"kubernetes.io/projected/41e60656-b3e9-4438-b73e-91e2632e452f-kube-api-access-qnl4d\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:23.723795 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:23.723758 4951 generic.go:356] "Generic (PLEG): container finished" podID="41e60656-b3e9-4438-b73e-91e2632e452f" containerID="44806be4f52b4ef13df780a24a249c52d8ab060c919c26ae359e10c2533aeb53" exitCode=0 Oct 30 07:26:23.724372 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:23.723811 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-58bf9bd7f9-7m5cm" Oct 30 07:26:23.724372 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:23.723839 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-58bf9bd7f9-7m5cm" event={"ID":"41e60656-b3e9-4438-b73e-91e2632e452f","Type":"ContainerDied","Data":"44806be4f52b4ef13df780a24a249c52d8ab060c919c26ae359e10c2533aeb53"} Oct 30 07:26:23.724372 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:23.723902 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-58bf9bd7f9-7m5cm" event={"ID":"41e60656-b3e9-4438-b73e-91e2632e452f","Type":"ContainerDied","Data":"d7a467277ccc782334418296c44ee16e2e346d6ca1bec3d50b530fd70dfec1b1"} Oct 30 07:26:23.724372 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:23.723937 4951 scope.go:122] "RemoveContainer" containerID="44806be4f52b4ef13df780a24a249c52d8ab060c919c26ae359e10c2533aeb53" Oct 30 07:26:23.725841 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:23.725813 4951 generic.go:356] "Generic (PLEG): container finished" podID="a133e6d0-be93-4023-9831-7dacca337709" containerID="7473c8c2a202f23e616dec718ac0d09fdd5983efaa5d706051ad8f72d9e0c97c" exitCode=0 Oct 30 07:26:23.725990 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:23.725874 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-8c45f755d-2tr2x" Oct 30 07:26:23.725990 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:23.725948 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-8c45f755d-2tr2x" event={"ID":"a133e6d0-be93-4023-9831-7dacca337709","Type":"ContainerDied","Data":"7473c8c2a202f23e616dec718ac0d09fdd5983efaa5d706051ad8f72d9e0c97c"} Oct 30 07:26:23.726107 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:23.726002 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-17bd11-8c45f755d-2tr2x" event={"ID":"a133e6d0-be93-4023-9831-7dacca337709","Type":"ContainerDied","Data":"2a763b059e4b9bc4e0cf220e63d83f9f835395bef5cc3d2aa25cccd5f2c2b481"} Oct 30 07:26:23.735390 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:23.735365 4951 scope.go:122] "RemoveContainer" containerID="44806be4f52b4ef13df780a24a249c52d8ab060c919c26ae359e10c2533aeb53" Oct 30 07:26:23.735795 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:26:23.735772 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44806be4f52b4ef13df780a24a249c52d8ab060c919c26ae359e10c2533aeb53\": container with ID starting with 44806be4f52b4ef13df780a24a249c52d8ab060c919c26ae359e10c2533aeb53 not found: ID does not exist" containerID="44806be4f52b4ef13df780a24a249c52d8ab060c919c26ae359e10c2533aeb53" Oct 30 07:26:23.735851 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:23.735805 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44806be4f52b4ef13df780a24a249c52d8ab060c919c26ae359e10c2533aeb53"} err="failed to get container status \"44806be4f52b4ef13df780a24a249c52d8ab060c919c26ae359e10c2533aeb53\": rpc error: code = NotFound desc = could not find container \"44806be4f52b4ef13df780a24a249c52d8ab060c919c26ae359e10c2533aeb53\": container with ID starting with 44806be4f52b4ef13df780a24a249c52d8ab060c919c26ae359e10c2533aeb53 not found: ID does not exist" Oct 30 07:26:23.735851 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:23.735824 4951 scope.go:122] "RemoveContainer" containerID="7473c8c2a202f23e616dec718ac0d09fdd5983efaa5d706051ad8f72d9e0c97c" Oct 30 07:26:23.749435 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:23.749402 4951 scope.go:122] "RemoveContainer" containerID="7473c8c2a202f23e616dec718ac0d09fdd5983efaa5d706051ad8f72d9e0c97c" Oct 30 07:26:23.749823 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:26:23.749799 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7473c8c2a202f23e616dec718ac0d09fdd5983efaa5d706051ad8f72d9e0c97c\": container with ID starting with 7473c8c2a202f23e616dec718ac0d09fdd5983efaa5d706051ad8f72d9e0c97c not found: ID does not exist" containerID="7473c8c2a202f23e616dec718ac0d09fdd5983efaa5d706051ad8f72d9e0c97c" Oct 30 07:26:23.749941 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:23.749830 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7473c8c2a202f23e616dec718ac0d09fdd5983efaa5d706051ad8f72d9e0c97c"} err="failed to get container status \"7473c8c2a202f23e616dec718ac0d09fdd5983efaa5d706051ad8f72d9e0c97c\": rpc error: code = NotFound desc = could not find container \"7473c8c2a202f23e616dec718ac0d09fdd5983efaa5d706051ad8f72d9e0c97c\": container with ID starting with 7473c8c2a202f23e616dec718ac0d09fdd5983efaa5d706051ad8f72d9e0c97c not found: ID does not exist" Oct 30 07:26:23.776388 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:23.776356 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-17bd11-8c45f755d-2tr2x"] Oct 30 07:26:23.784456 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:23.784423 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-17bd11-8c45f755d-2tr2x"] Oct 30 07:26:23.819509 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:23.819476 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-58bf9bd7f9-7m5cm"] Oct 30 07:26:23.824528 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:23.824499 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-17bd11-58bf9bd7f9-7m5cm"] Oct 30 07:26:25.781597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:25.780899 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41e60656-b3e9-4438-b73e-91e2632e452f" path="/var/lib/kubelet/pods/41e60656-b3e9-4438-b73e-91e2632e452f/volumes" Oct 30 07:26:25.782122 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:25.781720 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a133e6d0-be93-4023-9831-7dacca337709" path="/var/lib/kubelet/pods/a133e6d0-be93-4023-9831-7dacca337709/volumes" Oct 30 07:26:29.229075 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:29.229027 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk"] Oct 30 07:26:29.229550 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:29.229313 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" podUID="f1d68158-e979-4969-9918-e75a33ec299a" containerName="istio-proxy" containerID="cri-o://2e300a806dbdbbc7cc5770848bbbfc367f514e3fdce4df592cd3c21dc9e51a10" gracePeriod=30 Oct 30 07:26:34.234697 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.234655 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/3.log" Oct 30 07:26:34.238125 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.238094 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/3.log" Oct 30 07:26:34.242410 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.242375 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 07:26:34.245406 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.245382 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 07:26:34.558949 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.558917 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:26:34.620882 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.620847 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-workload-socket\") pod \"f1d68158-e979-4969-9918-e75a33ec299a\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " Oct 30 07:26:34.621044 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.620921 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1d68158-e979-4969-9918-e75a33ec299a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1d68158-e979-4969-9918-e75a33ec299a-istio-podinfo\") pod \"f1d68158-e979-4969-9918-e75a33ec299a\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " Oct 30 07:26:34.621044 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.620956 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-istio-envoy\") pod \"f1d68158-e979-4969-9918-e75a33ec299a\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " Oct 30 07:26:34.621044 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.621002 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1d68158-e979-4969-9918-e75a33ec299a-kube-api-access-f6dcd\" (UniqueName: \"kubernetes.io/projected/f1d68158-e979-4969-9918-e75a33ec299a-kube-api-access-f6dcd\") pod \"f1d68158-e979-4969-9918-e75a33ec299a\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " Oct 30 07:26:34.621246 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.621056 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1d68158-e979-4969-9918-e75a33ec299a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1d68158-e979-4969-9918-e75a33ec299a-istiod-ca-cert\") pod \"f1d68158-e979-4969-9918-e75a33ec299a\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " Oct 30 07:26:34.621246 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.621096 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1d68158-e979-4969-9918-e75a33ec299a-istio-token\" (UniqueName: \"kubernetes.io/projected/f1d68158-e979-4969-9918-e75a33ec299a-istio-token\") pod \"f1d68158-e979-4969-9918-e75a33ec299a\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " Oct 30 07:26:34.621246 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.621111 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-workload-socket" pod "f1d68158-e979-4969-9918-e75a33ec299a" (UID: "f1d68158-e979-4969-9918-e75a33ec299a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:34.621246 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.621136 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-workload-certs\") pod \"f1d68158-e979-4969-9918-e75a33ec299a\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " Oct 30 07:26:34.621246 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.621246 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-credential-socket\") pod \"f1d68158-e979-4969-9918-e75a33ec299a\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " Oct 30 07:26:34.621519 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.621282 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-istio-data\") pod \"f1d68158-e979-4969-9918-e75a33ec299a\" (UID: \"f1d68158-e979-4969-9918-e75a33ec299a\") " Oct 30 07:26:34.621838 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.621808 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:34.622876 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.622445 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-credential-socket" pod "f1d68158-e979-4969-9918-e75a33ec299a" (UID: "f1d68158-e979-4969-9918-e75a33ec299a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:34.622876 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.622676 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-workload-certs" pod "f1d68158-e979-4969-9918-e75a33ec299a" (UID: "f1d68158-e979-4969-9918-e75a33ec299a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:34.622876 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.622724 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-istio-data" pod "f1d68158-e979-4969-9918-e75a33ec299a" (UID: "f1d68158-e979-4969-9918-e75a33ec299a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:34.622876 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.622838 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1d68158-e979-4969-9918-e75a33ec299a-istiod-ca-cert" pod "f1d68158-e979-4969-9918-e75a33ec299a" (UID: "f1d68158-e979-4969-9918-e75a33ec299a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:26:34.625121 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.625100 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1d68158-e979-4969-9918-e75a33ec299a-istio-podinfo" pod "f1d68158-e979-4969-9918-e75a33ec299a" (UID: "f1d68158-e979-4969-9918-e75a33ec299a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:26:34.625407 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.625388 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-istio-envoy" pod "f1d68158-e979-4969-9918-e75a33ec299a" (UID: "f1d68158-e979-4969-9918-e75a33ec299a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:34.625517 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.625502 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d68158-e979-4969-9918-e75a33ec299a-kube-api-access-f6dcd" pod "f1d68158-e979-4969-9918-e75a33ec299a" (UID: "f1d68158-e979-4969-9918-e75a33ec299a"). InnerVolumeSpecName "kube-api-access-f6dcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:26:34.625739 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.625701 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d68158-e979-4969-9918-e75a33ec299a-istio-token" pod "f1d68158-e979-4969-9918-e75a33ec299a" (UID: "f1d68158-e979-4969-9918-e75a33ec299a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:26:34.723263 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.723235 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:34.723263 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.723262 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:34.723442 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.723274 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1d68158-e979-4969-9918-e75a33ec299a-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:34.723442 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.723282 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:34.723442 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.723291 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6dcd\" (UniqueName: \"kubernetes.io/projected/f1d68158-e979-4969-9918-e75a33ec299a-kube-api-access-f6dcd\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:34.723442 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.723300 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1d68158-e979-4969-9918-e75a33ec299a-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:34.723442 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.723308 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1d68158-e979-4969-9918-e75a33ec299a-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:34.723442 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.723316 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1d68158-e979-4969-9918-e75a33ec299a-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:26:34.807780 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.807696 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4a665-c894f7bc5-ccjfc" Oct 30 07:26:34.886740 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.886706 4951 generic.go:356] "Generic (PLEG): container finished" podID="f1d68158-e979-4969-9918-e75a33ec299a" containerID="2e300a806dbdbbc7cc5770848bbbfc367f514e3fdce4df592cd3c21dc9e51a10" exitCode=0 Oct 30 07:26:34.887110 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.886944 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" event={"ID":"f1d68158-e979-4969-9918-e75a33ec299a","Type":"ContainerDied","Data":"2e300a806dbdbbc7cc5770848bbbfc367f514e3fdce4df592cd3c21dc9e51a10"} Oct 30 07:26:34.887246 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.887225 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" event={"ID":"f1d68158-e979-4969-9918-e75a33ec299a","Type":"ContainerDied","Data":"7fe9e45aafb64b75e0213b5e7ded98ec1b5d74478ceceb4d3b83c5c989d82fbb"} Oct 30 07:26:34.887317 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.887268 4951 scope.go:122] "RemoveContainer" containerID="2e300a806dbdbbc7cc5770848bbbfc367f514e3fdce4df592cd3c21dc9e51a10" Oct 30 07:26:34.887393 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.886991 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk" Oct 30 07:26:34.900600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.900464 4951 scope.go:122] "RemoveContainer" containerID="2e300a806dbdbbc7cc5770848bbbfc367f514e3fdce4df592cd3c21dc9e51a10" Oct 30 07:26:34.901830 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:26:34.901510 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e300a806dbdbbc7cc5770848bbbfc367f514e3fdce4df592cd3c21dc9e51a10\": container with ID starting with 2e300a806dbdbbc7cc5770848bbbfc367f514e3fdce4df592cd3c21dc9e51a10 not found: ID does not exist" containerID="2e300a806dbdbbc7cc5770848bbbfc367f514e3fdce4df592cd3c21dc9e51a10" Oct 30 07:26:34.901830 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.901567 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e300a806dbdbbc7cc5770848bbbfc367f514e3fdce4df592cd3c21dc9e51a10"} err="failed to get container status \"2e300a806dbdbbc7cc5770848bbbfc367f514e3fdce4df592cd3c21dc9e51a10\": rpc error: code = NotFound desc = could not find container \"2e300a806dbdbbc7cc5770848bbbfc367f514e3fdce4df592cd3c21dc9e51a10\": container with ID starting with 2e300a806dbdbbc7cc5770848bbbfc367f514e3fdce4df592cd3c21dc9e51a10 not found: ID does not exist" Oct 30 07:26:34.943632 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.943601 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk"] Oct 30 07:26:34.947927 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:34.947896 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bdtxk"] Oct 30 07:26:35.074925 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:35.074830 4951 scope.go:122] "RemoveContainer" containerID="b423a31dfcf62a0367e1bf8b845219f6c60551a376e21cf9d8d4de5e80294222" Oct 30 07:26:35.772331 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:35.772294 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1d68158-e979-4969-9918-e75a33ec299a" path="/var/lib/kubelet/pods/f1d68158-e979-4969-9918-e75a33ec299a/volumes" Oct 30 07:26:37.810148 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:37.810114 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d"] Oct 30 07:26:37.810876 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:37.810850 4951 cpu_manager.go:414] "containerMap: removing container" podUID="a133e6d0-be93-4023-9831-7dacca337709" containerName="payload-processing" Oct 30 07:26:37.810876 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:37.810877 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a133e6d0-be93-4023-9831-7dacca337709" containerName="payload-processing" Oct 30 07:26:37.811092 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:37.810910 4951 cpu_manager.go:414] "containerMap: removing container" podUID="41e60656-b3e9-4438-b73e-91e2632e452f" containerName="payload-pre-processing" Oct 30 07:26:37.811092 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:37.810917 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41e60656-b3e9-4438-b73e-91e2632e452f" containerName="payload-pre-processing" Oct 30 07:26:37.811092 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:37.810925 4951 cpu_manager.go:414] "containerMap: removing container" podUID="f1d68158-e979-4969-9918-e75a33ec299a" containerName="istio-proxy" Oct 30 07:26:37.811092 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:37.810932 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1d68158-e979-4969-9918-e75a33ec299a" containerName="istio-proxy" Oct 30 07:26:37.811092 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:37.811054 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="41e60656-b3e9-4438-b73e-91e2632e452f" containerName="payload-pre-processing" Oct 30 07:26:37.811092 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:37.811066 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1d68158-e979-4969-9918-e75a33ec299a" containerName="istio-proxy" Oct 30 07:26:37.811092 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:37.811081 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="a133e6d0-be93-4023-9831-7dacca337709" containerName="payload-processing" Oct 30 07:26:37.816880 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:37.816861 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d" Oct 30 07:26:37.837780 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:37.837748 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-f4a665-serving-cert\"" Oct 30 07:26:37.846268 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:37.846229 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d"] Oct 30 07:26:37.863109 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:37.863067 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0149e3ab-1dff-46b7-acef-d557a1bb83e0-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d\" (UID: \"0149e3ab-1dff-46b7-acef-d557a1bb83e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d" Oct 30 07:26:37.863311 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:37.863271 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptsdn\" (UniqueName: \"kubernetes.io/projected/0149e3ab-1dff-46b7-acef-d557a1bb83e0-kube-api-access-ptsdn\") pod \"maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d\" (UID: \"0149e3ab-1dff-46b7-acef-d557a1bb83e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d" Oct 30 07:26:37.863529 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:37.863502 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0149e3ab-1dff-46b7-acef-d557a1bb83e0-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d\" (UID: \"0149e3ab-1dff-46b7-acef-d557a1bb83e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d" Oct 30 07:26:37.863686 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:37.863666 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0149e3ab-1dff-46b7-acef-d557a1bb83e0-metrics-certs\") pod \"maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d\" (UID: \"0149e3ab-1dff-46b7-acef-d557a1bb83e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d" Oct 30 07:26:37.964832 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:37.964793 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0149e3ab-1dff-46b7-acef-d557a1bb83e0-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d\" (UID: \"0149e3ab-1dff-46b7-acef-d557a1bb83e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d" Oct 30 07:26:37.965038 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:37.964882 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptsdn\" (UniqueName: \"kubernetes.io/projected/0149e3ab-1dff-46b7-acef-d557a1bb83e0-kube-api-access-ptsdn\") pod \"maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d\" (UID: \"0149e3ab-1dff-46b7-acef-d557a1bb83e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d" Oct 30 07:26:37.965127 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:37.965034 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0149e3ab-1dff-46b7-acef-d557a1bb83e0-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d\" (UID: \"0149e3ab-1dff-46b7-acef-d557a1bb83e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d" Oct 30 07:26:37.965185 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:37.965136 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0149e3ab-1dff-46b7-acef-d557a1bb83e0-metrics-certs\") pod \"maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d\" (UID: \"0149e3ab-1dff-46b7-acef-d557a1bb83e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d" Oct 30 07:26:37.965687 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:37.965662 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0149e3ab-1dff-46b7-acef-d557a1bb83e0-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d\" (UID: \"0149e3ab-1dff-46b7-acef-d557a1bb83e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d" Oct 30 07:26:37.968090 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:37.968069 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0149e3ab-1dff-46b7-acef-d557a1bb83e0-metrics-certs\") pod \"maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d\" (UID: \"0149e3ab-1dff-46b7-acef-d557a1bb83e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d" Oct 30 07:26:37.968296 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:37.968276 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0149e3ab-1dff-46b7-acef-d557a1bb83e0-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d\" (UID: \"0149e3ab-1dff-46b7-acef-d557a1bb83e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d" Oct 30 07:26:37.988655 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:37.988624 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptsdn\" (UniqueName: \"kubernetes.io/projected/0149e3ab-1dff-46b7-acef-d557a1bb83e0-kube-api-access-ptsdn\") pod \"maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d\" (UID: \"0149e3ab-1dff-46b7-acef-d557a1bb83e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d" Oct 30 07:26:38.131747 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:38.131660 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d" Oct 30 07:26:38.440452 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:38.440422 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d"] Oct 30 07:26:38.441137 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:26:38.441111 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0149e3ab_1dff_46b7_acef_d557a1bb83e0.slice/crio-bad24772c9b336a53beb4c8e530a5152a51e4f99b95be586070387a8e0bba3ed WatchSource:0}: Error finding container bad24772c9b336a53beb4c8e530a5152a51e4f99b95be586070387a8e0bba3ed: Status 404 returned error can't find the container with id bad24772c9b336a53beb4c8e530a5152a51e4f99b95be586070387a8e0bba3ed Oct 30 07:26:38.917249 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:38.917201 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d" event={"ID":"0149e3ab-1dff-46b7-acef-d557a1bb83e0","Type":"ContainerStarted","Data":"db7d72fbbe9443301e77e5e3e264f340ab0c2546b16fad60736501bfeb0f3ac1"} Oct 30 07:26:38.917841 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:38.917278 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d" Oct 30 07:26:38.917841 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:38.917298 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d" event={"ID":"0149e3ab-1dff-46b7-acef-d557a1bb83e0","Type":"ContainerStarted","Data":"bad24772c9b336a53beb4c8e530a5152a51e4f99b95be586070387a8e0bba3ed"} Oct 30 07:26:38.961913 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:38.961850 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d" podStartSLOduration=1.58341288 podStartE2EDuration="1.961830938s" totalImagesPullingTime="378.418058ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:38.960207433 +0000 UTC m=+1805.303255397" watchObservedRunningTime="2026-10-30 07:26:38.961830938 +0000 UTC m=+1805.304878908" Oct 30 07:26:44.931482 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:44.931416 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d" Oct 30 07:26:45.595553 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:45.595519 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7"] Oct 30 07:26:45.600008 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:45.599982 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:26:45.634705 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:45.634666 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7"] Oct 30 07:26:45.692501 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:45.692463 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:26:45.692745 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:45.692521 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:26:45.692745 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:45.692639 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:26:45.692745 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:45.692669 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76nhv\" (UniqueName: \"kubernetes.io/projected/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-kube-api-access-76nhv\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:26:45.692745 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:45.692698 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:26:45.692931 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:45.692826 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:26:45.692999 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:45.692978 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:26:45.693040 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:45.693027 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:26:45.693110 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:45.693098 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:26:45.794708 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:45.794675 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:26:45.794894 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:45.794740 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:26:45.794894 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:45.794786 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:26:45.794894 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:45.794817 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:26:45.794894 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:45.794854 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:26:45.794894 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:45.794890 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:26:45.795205 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:45.794929 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76nhv\" (UniqueName: \"kubernetes.io/projected/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-kube-api-access-76nhv\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:26:45.795205 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:45.794984 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:26:45.795205 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:45.795037 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:26:45.795351 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:45.795206 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:26:45.795351 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:45.795245 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:26:45.795429 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:45.795373 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:26:45.795472 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:45.795427 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:26:45.795653 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:45.795631 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:26:45.797313 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:45.797289 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:26:45.797422 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:45.797406 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:26:45.818313 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:45.818283 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76nhv\" (UniqueName: \"kubernetes.io/projected/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-kube-api-access-76nhv\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:26:45.818473 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:45.818447 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:26:45.915988 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:45.915917 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:26:46.246698 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:46.243712 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7"] Oct 30 07:26:46.252727 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:46.252686 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:26:46.252853 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:46.252781 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:26:46.252853 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:46.252821 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:26:46.918523 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:46.917547 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8"] Oct 30 07:26:46.937372 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:46.936460 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts"] Oct 30 07:26:46.938768 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:46.938736 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:26:46.941129 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:46.941089 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:26:46.951269 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:46.951230 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8"] Oct 30 07:26:46.954310 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:46.954283 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e5fa2d-openshift-default-dockercfg-sppbj\"" Oct 30 07:26:46.983903 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:46.983794 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts"] Oct 30 07:26:46.996061 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:46.996022 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" event={"ID":"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5","Type":"ContainerStarted","Data":"253c9c2ae58366494085b687f9c461607d701ff9ed9d6cf8b0cce3df4f51ea87"} Oct 30 07:26:46.996216 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:46.996091 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" event={"ID":"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5","Type":"ContainerStarted","Data":"20c78162cb20f4c5a5414e2ac04cbeb94d3727826342e7037e1b6e02e05ff623"} Oct 30 07:26:47.075874 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.075814 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" podStartSLOduration=2.075795592 podStartE2EDuration="2.075795592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:47.072367552 +0000 UTC m=+1813.415415517" watchObservedRunningTime="2026-10-30 07:26:47.075795592 +0000 UTC m=+1813.418843558" Oct 30 07:26:47.123427 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.123383 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:26:47.123624 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.123450 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:26:47.123624 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.123527 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:26:47.123624 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.123559 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:26:47.123801 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.123670 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:26:47.123801 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.123699 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246b0113-0e73-46b7-88f6-3fac16df57b5-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:26:47.123801 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.123768 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246b0113-0e73-46b7-88f6-3fac16df57b5-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:26:47.123801 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.123793 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:26:47.124013 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.123832 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea726207-9a8b-4494-bf06-c339b4114002-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:26:47.124013 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.123866 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pftfx\" (UniqueName: \"kubernetes.io/projected/ea726207-9a8b-4494-bf06-c339b4114002-kube-api-access-pftfx\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:26:47.124134 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.124063 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:26:47.124191 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.124151 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea726207-9a8b-4494-bf06-c339b4114002-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:26:47.124246 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.124191 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:26:47.124327 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.124307 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea726207-9a8b-4494-bf06-c339b4114002-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:26:47.124389 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.124364 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wc6t\" (UniqueName: \"kubernetes.io/projected/246b0113-0e73-46b7-88f6-3fac16df57b5-kube-api-access-6wc6t\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:26:47.124448 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.124402 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246b0113-0e73-46b7-88f6-3fac16df57b5-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:26:47.124543 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.124526 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:26:47.124624 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.124603 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:26:47.232646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.226352 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:26:47.232646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.226425 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea726207-9a8b-4494-bf06-c339b4114002-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:26:47.232646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.226456 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:26:47.232646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.226513 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea726207-9a8b-4494-bf06-c339b4114002-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:26:47.232646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.226557 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wc6t\" (UniqueName: \"kubernetes.io/projected/246b0113-0e73-46b7-88f6-3fac16df57b5-kube-api-access-6wc6t\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:26:47.232646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.226609 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246b0113-0e73-46b7-88f6-3fac16df57b5-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:26:47.232646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.226652 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:26:47.232646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.226706 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:26:47.232646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.226797 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:26:47.232646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.226849 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:26:47.232646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.226875 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:26:47.232646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.226902 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:26:47.232646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.226943 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:26:47.232646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.226968 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246b0113-0e73-46b7-88f6-3fac16df57b5-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:26:47.232646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.227035 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246b0113-0e73-46b7-88f6-3fac16df57b5-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:26:47.232646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.227067 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:26:47.232646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.227104 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea726207-9a8b-4494-bf06-c339b4114002-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:26:47.232646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.227130 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pftfx\" (UniqueName: \"kubernetes.io/projected/ea726207-9a8b-4494-bf06-c339b4114002-kube-api-access-pftfx\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:26:47.232646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.227962 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:26:47.232646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.228756 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:26:47.232646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.228992 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:26:47.237593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.234953 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:26:47.237593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.235201 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246b0113-0e73-46b7-88f6-3fac16df57b5-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:26:47.237593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.235216 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:26:47.237593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.235456 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:26:47.237593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.235907 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:26:47.240976 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.240228 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:26:47.240976 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.240923 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea726207-9a8b-4494-bf06-c339b4114002-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:26:47.253909 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.253866 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:26:47.255120 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.254426 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea726207-9a8b-4494-bf06-c339b4114002-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:26:47.255120 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.255067 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pftfx\" (UniqueName: \"kubernetes.io/projected/ea726207-9a8b-4494-bf06-c339b4114002-kube-api-access-pftfx\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:26:47.264700 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.264474 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246b0113-0e73-46b7-88f6-3fac16df57b5-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:26:47.265212 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.265179 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wc6t\" (UniqueName: \"kubernetes.io/projected/246b0113-0e73-46b7-88f6-3fac16df57b5-kube-api-access-6wc6t\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:26:47.265942 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.265916 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea726207-9a8b-4494-bf06-c339b4114002-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:26:47.266859 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.266837 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:26:47.268959 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.268927 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246b0113-0e73-46b7-88f6-3fac16df57b5-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:26:47.284111 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.284091 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:26:47.297211 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.297194 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:26:47.658044 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.657736 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8"] Oct 30 07:26:47.662516 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.662492 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts"] Oct 30 07:26:47.662940 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:26:47.662908 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea726207_9a8b_4494_bf06_c339b4114002.slice/crio-492535d95211181d56fc6842fc8c644838b702529ccad48c6d96eba41e75f616 WatchSource:0}: Error finding container 492535d95211181d56fc6842fc8c644838b702529ccad48c6d96eba41e75f616: Status 404 returned error can't find the container with id 492535d95211181d56fc6842fc8c644838b702529ccad48c6d96eba41e75f616 Oct 30 07:26:47.663801 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:26:47.663784 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod246b0113_0e73_46b7_88f6_3fac16df57b5.slice/crio-e50e88607dcf8b6c904dcc9ce151fa748f70c6625a42072308fbc014d6b5b53f WatchSource:0}: Error finding container e50e88607dcf8b6c904dcc9ce151fa748f70c6625a42072308fbc014d6b5b53f: Status 404 returned error can't find the container with id e50e88607dcf8b6c904dcc9ce151fa748f70c6625a42072308fbc014d6b5b53f Oct 30 07:26:47.666709 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.666679 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:26:47.666906 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.666880 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:26:47.667015 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.666989 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:26:47.667064 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.667052 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:26:47.667098 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.667081 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:26:47.667157 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.667140 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:26:47.918733 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.918621 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:26:47.919778 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.919707 4951 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 07:26:47.919778 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:47.919752 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" podUID="9ba10610-90eb-43ad-b937-e6c4d4a9dfb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 07:26:48.012249 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:48.007683 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" event={"ID":"246b0113-0e73-46b7-88f6-3fac16df57b5","Type":"ContainerStarted","Data":"7ec5f74bbbe481bfece2ca84efd2eb85dda0ae5eb67b577b439a8ea135d5dd6f"} Oct 30 07:26:48.012249 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:48.007749 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" event={"ID":"246b0113-0e73-46b7-88f6-3fac16df57b5","Type":"ContainerStarted","Data":"e50e88607dcf8b6c904dcc9ce151fa748f70c6625a42072308fbc014d6b5b53f"} Oct 30 07:26:48.012249 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:48.011506 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" event={"ID":"ea726207-9a8b-4494-bf06-c339b4114002","Type":"ContainerStarted","Data":"0ad8985d2330ab47dc9a23f7e96c28fbe1c7ca007df288947f58d25c90b4e078"} Oct 30 07:26:48.012249 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:48.011560 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" event={"ID":"ea726207-9a8b-4494-bf06-c339b4114002","Type":"ContainerStarted","Data":"492535d95211181d56fc6842fc8c644838b702529ccad48c6d96eba41e75f616"} Oct 30 07:26:48.097471 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:48.096980 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" podStartSLOduration=2.096960998 podStartE2EDuration="2.096960998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:48.094289125 +0000 UTC m=+1814.437337091" watchObservedRunningTime="2026-10-30 07:26:48.096960998 +0000 UTC m=+1814.440008968" Oct 30 07:26:48.170661 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:48.163014 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" podStartSLOduration=2.162991845 podStartE2EDuration="2.162991845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:48.145896901 +0000 UTC m=+1814.488944880" watchObservedRunningTime="2026-10-30 07:26:48.162991845 +0000 UTC m=+1814.506039812" Oct 30 07:26:48.285392 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:48.285358 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:26:48.287293 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:48.287263 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 07:26:48.287450 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:48.287322 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" podUID="ea726207-9a8b-4494-bf06-c339b4114002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 07:26:48.297865 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:48.297609 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:26:48.298900 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:48.298871 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 07:26:48.298991 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:48.298935 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" podUID="246b0113-0e73-46b7-88f6-3fac16df57b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 07:26:48.931752 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:48.931712 4951 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 07:26:48.932409 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:48.931998 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" podUID="9ba10610-90eb-43ad-b937-e6c4d4a9dfb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 07:26:49.295596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:49.295254 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 07:26:49.295596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:49.295321 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" podUID="ea726207-9a8b-4494-bf06-c339b4114002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 07:26:49.308208 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:49.308172 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 07:26:49.309605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:49.309552 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" podUID="246b0113-0e73-46b7-88f6-3fac16df57b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 07:26:49.923626 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:49.923587 4951 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 07:26:49.979597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:49.958058 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" podUID="9ba10610-90eb-43ad-b937-e6c4d4a9dfb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 07:26:50.296400 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:50.295720 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 07:26:50.296400 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:50.295786 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" podUID="ea726207-9a8b-4494-bf06-c339b4114002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 07:26:50.297902 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:50.297801 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 07:26:50.297902 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:50.297852 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" podUID="246b0113-0e73-46b7-88f6-3fac16df57b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 07:26:50.925596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:50.920037 4951 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 07:26:50.925596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:50.920105 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" podUID="9ba10610-90eb-43ad-b937-e6c4d4a9dfb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 07:26:51.288028 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:51.287988 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 07:26:51.289038 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:51.289007 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" podUID="ea726207-9a8b-4494-bf06-c339b4114002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 07:26:51.315469 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:51.315437 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 07:26:51.316453 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:51.316424 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" podUID="246b0113-0e73-46b7-88f6-3fac16df57b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 07:26:51.373679 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:51.373635 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 07:26:51.394871 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:51.394826 4951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded" Oct 30 07:26:51.592843 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:51.592727 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:51.679737 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:51.679691 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" podUID="8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.168:8000/ready\": context deadline exceeded" Oct 30 07:26:51.811923 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:51.810692 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:51.920913 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:51.920814 4951 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 07:26:51.921675 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:51.921151 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" podUID="9ba10610-90eb-43ad-b937-e6c4d4a9dfb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 07:26:51.927684 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:51.927638 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:51.968199 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:51.968156 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:52.319108 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:52.319067 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 07:26:52.384982 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:52.382816 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:52.385712 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:52.353355 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 07:26:52.385712 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:52.385658 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" podUID="246b0113-0e73-46b7-88f6-3fac16df57b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 07:26:52.417610 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:52.385982 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:26:52.419605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:52.419542 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" podUID="ea726207-9a8b-4494-bf06-c339b4114002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 07:26:52.636598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:52.636487 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:52.637139 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:52.637118 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:26:52.694748 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:52.686429 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" podUID="8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.168:8000/ready\": context deadline exceeded" Oct 30 07:26:52.694748 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:52.689810 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" Oct 30 07:26:52.821600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:52.820739 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:52.821600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:52.821247 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:26:53.021608 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:53.015846 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:53.080215 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:53.080150 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 07:26:53.086611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:53.067197 4951 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:26:53.086611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:53.081713 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" podUID="d493c768-747e-40e3-8e65-1c7fcb582c23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:53.086611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:53.083794 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:26:53.086611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:53.083865 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:26:53.141843 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:53.141807 4951 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 07:26:53.142115 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:53.142073 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" podUID="9ba10610-90eb-43ad-b937-e6c4d4a9dfb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 07:26:53.324059 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:53.323965 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 07:26:53.328690 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:53.328641 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" podUID="246b0113-0e73-46b7-88f6-3fac16df57b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 07:26:53.363680 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:53.349667 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 07:26:53.513596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:53.513014 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" podUID="ea726207-9a8b-4494-bf06-c339b4114002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 07:26:53.626650 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:53.621760 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:53.689710 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:53.689666 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:53.753642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:53.704849 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" podUID="8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.168:8000/ready\": context deadline exceeded" Oct 30 07:26:53.760683 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:53.760632 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:53.885664 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:53.822984 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:53.918003 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:26:53.916795 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:26:53.918003 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:53.917897 4951 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 07:26:53.918003 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:53.917945 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" podUID="9ba10610-90eb-43ad-b937-e6c4d4a9dfb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 07:26:54.049609 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:54.043246 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:26:54.103597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:54.089494 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:54.293604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:54.293537 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 07:26:54.293817 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:54.293613 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" podUID="ea726207-9a8b-4494-bf06-c339b4114002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 07:26:54.374262 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:54.373686 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 07:26:54.374262 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:54.373751 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" podUID="246b0113-0e73-46b7-88f6-3fac16df57b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 07:26:54.623653 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:54.622752 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:54.727595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:54.724676 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" podUID="8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.168:8000/ready\": context deadline exceeded" Oct 30 07:26:54.727595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:54.725110 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 07:26:54.798028 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:54.797981 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:54.876389 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:54.876282 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:54.890959 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:54.877860 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:54.891594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:54.891554 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:26:54.978558 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:54.978519 4951 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 07:26:54.978880 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:54.978845 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" podUID="9ba10610-90eb-43ad-b937-e6c4d4a9dfb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 07:26:55.040847 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:55.040788 4951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:55.088965 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:55.088914 4951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:55.090800 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:55.090765 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 07:26:55.279707 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:55.274091 4951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:55.321466 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:55.321429 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 07:26:55.321744 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:55.321713 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" podUID="ea726207-9a8b-4494-bf06-c339b4114002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 07:26:55.345593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:55.329420 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 07:26:55.345593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:55.329469 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" podUID="246b0113-0e73-46b7-88f6-3fac16df57b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 07:26:55.641655 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:55.626719 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:55.732718 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:55.732669 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" podUID="8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.168:8000/ready\": context deadline exceeded" Oct 30 07:26:55.733307 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:55.732669 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:55.842598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:55.824941 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 07:26:55.899039 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:55.894488 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:55.899039 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:55.894941 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 07:26:55.918936 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:55.918902 4951 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 07:26:55.919180 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:55.919152 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" podUID="9ba10610-90eb-43ad-b937-e6c4d4a9dfb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 07:26:56.097598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:56.093680 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:56.097598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:56.095832 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:26:56.280635 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:56.279329 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:26:56.314595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:56.300045 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 07:26:56.314595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:56.300108 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" podUID="ea726207-9a8b-4494-bf06-c339b4114002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 07:26:56.335604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:56.335290 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 07:26:56.335604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:56.335350 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" podUID="246b0113-0e73-46b7-88f6-3fac16df57b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 07:26:56.650420 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:56.650318 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:56.733236 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:56.733189 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" podUID="8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.168:8000/ready\": context deadline exceeded" Oct 30 07:26:56.734303 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:56.734268 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:56.857596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:56.856950 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:56.909806 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:56.909702 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:56.931891 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:56.931859 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:26:57.132136 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:57.132101 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:26:57.301609 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:57.301556 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 07:26:57.302394 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:57.302366 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" podUID="246b0113-0e73-46b7-88f6-3fac16df57b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 07:26:57.307421 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:57.302225 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 07:26:57.307872 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:57.302299 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 07:26:57.314660 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:57.314628 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" podUID="ea726207-9a8b-4494-bf06-c339b4114002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 07:26:57.402736 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:57.402681 4951 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" podUID="8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.168:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:57.505281 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:57.505246 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:26:57.538715 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:57.538681 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:26:57.683658 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:57.669819 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:57.749707 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:57.749658 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" podUID="8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.168:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:57.911171 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:57.911125 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:26:57.916103 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:57.916071 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" Oct 30 07:26:57.919387 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:57.919351 4951 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:26:57.919596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:57.919409 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" podUID="9ba10610-90eb-43ad-b937-e6c4d4a9dfb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:57.990668 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:57.988538 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:26:57.990668 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:57.989551 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:26:57.990925 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:57.990745 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:26:58.285305 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:58.285266 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 07:26:58.291931 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:58.291880 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" podUID="ea726207-9a8b-4494-bf06-c339b4114002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 07:26:58.319254 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:58.319126 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 07:26:58.319254 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:58.319184 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" podUID="246b0113-0e73-46b7-88f6-3fac16df57b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 07:26:59.570593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:59.569770 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:26:59.574591 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:59.571274 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:26:59.574591 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:26:59.572662 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:27:00.341814 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:00.341726 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:27:00.341814 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:00.341796 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" podUID="246b0113-0e73-46b7-88f6-3fac16df57b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:01.283516 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:01.282149 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:27:01.283516 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:01.283384 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:27:01.285648 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:01.285595 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:27:01.999598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:01.981526 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4"] Oct 30 07:27:02.006406 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.006376 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4"] Oct 30 07:27:02.006600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.006484 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:27:02.037891 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.037828 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:27:02.037891 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.037888 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:27:02.038103 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.037955 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/088ac46a-3235-420e-9680-74cb738f229c-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:27:02.038103 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.038080 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:27:02.038234 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.038137 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/088ac46a-3235-420e-9680-74cb738f229c-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:27:02.038234 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.038187 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkxlk\" (UniqueName: \"kubernetes.io/projected/088ac46a-3235-420e-9680-74cb738f229c-kube-api-access-bkxlk\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:27:02.038401 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.038252 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:27:02.038401 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.038277 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:27:02.038401 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.038312 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/088ac46a-3235-420e-9680-74cb738f229c-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:27:02.139474 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.139439 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:27:02.139697 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.139511 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/088ac46a-3235-420e-9680-74cb738f229c-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:27:02.139697 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.139584 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkxlk\" (UniqueName: \"kubernetes.io/projected/088ac46a-3235-420e-9680-74cb738f229c-kube-api-access-bkxlk\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:27:02.139697 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.139664 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:27:02.139697 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.139688 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:27:02.139921 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.139725 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/088ac46a-3235-420e-9680-74cb738f229c-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:27:02.139921 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.139803 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:27:02.139921 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.139842 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:27:02.139921 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.139893 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/088ac46a-3235-420e-9680-74cb738f229c-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:27:02.140268 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.140238 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:27:02.140498 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.140452 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:27:02.140601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.140510 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:27:02.140840 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.140818 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:27:02.141134 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.141106 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/088ac46a-3235-420e-9680-74cb738f229c-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:27:02.143091 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.143065 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/088ac46a-3235-420e-9680-74cb738f229c-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:27:02.146835 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.146809 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:27:02.163769 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.163734 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/088ac46a-3235-420e-9680-74cb738f229c-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:27:02.164421 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.164396 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkxlk\" (UniqueName: \"kubernetes.io/projected/088ac46a-3235-420e-9680-74cb738f229c-kube-api-access-bkxlk\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:27:02.334875 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.334787 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:27:02.662260 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.662226 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4"] Oct 30 07:27:02.664561 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:27:02.664531 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod088ac46a_3235_420e_9680_74cb738f229c.slice/crio-18b16e180063ed1e9ace0989030902816548352c860e7031b000da51942017cb WatchSource:0}: Error finding container 18b16e180063ed1e9ace0989030902816548352c860e7031b000da51942017cb: Status 404 returned error can't find the container with id 18b16e180063ed1e9ace0989030902816548352c860e7031b000da51942017cb Oct 30 07:27:02.668316 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.668270 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:27:02.668435 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.668374 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:27:02.668504 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.668444 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:27:02.770611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.767325 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" event={"ID":"088ac46a-3235-420e-9680-74cb738f229c","Type":"ContainerStarted","Data":"5d6cc1f3ab5a51c8b8ccae89eaba690e43c909b9685a7033707cc3728f228a35"} Oct 30 07:27:02.770611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.767385 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" event={"ID":"088ac46a-3235-420e-9680-74cb738f229c","Type":"ContainerStarted","Data":"18b16e180063ed1e9ace0989030902816548352c860e7031b000da51942017cb"} Oct 30 07:27:02.811724 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:02.811673 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" podStartSLOduration=1.811659975 podStartE2EDuration="1.811659975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:27:02.811002782 +0000 UTC m=+1829.154050784" watchObservedRunningTime="2026-10-30 07:27:02.811659975 +0000 UTC m=+1829.154707939" Oct 30 07:27:03.345606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:03.338129 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:27:03.359651 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:03.351438 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 07:27:03.359651 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:03.351500 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" podUID="088ac46a-3235-420e-9680-74cb738f229c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 07:27:03.596832 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:03.596742 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv"] Oct 30 07:27:03.597038 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:03.596996 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" podUID="b0938f71-f1d8-4342-8d0a-7aed59adef39" containerName="istio-proxy" containerID="cri-o://c30a4cba4d5a0410ce429bd419852e14580de8fe34bc5b07bd2e1a149718e385" gracePeriod=30 Oct 30 07:27:04.405079 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:04.404585 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 07:27:04.405079 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:04.404658 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" podUID="088ac46a-3235-420e-9680-74cb738f229c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 07:27:04.616191 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:04.614082 4951 patch_prober.go:29] interesting pod/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": read tcp 10.132.0.2:52464->10.132.0.169:15021: read: connection reset by peer" start-of-body= Oct 30 07:27:04.616191 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:04.614159 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" podUID="0e446711-8b80-43b4-9c61-b69b3f8a320b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": read tcp 10.132.0.2:52464->10.132.0.169:15021: read: connection reset by peer" Oct 30 07:27:04.919730 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:04.919685 4951 generic.go:356] "Generic (PLEG): container finished" podID="0e446711-8b80-43b4-9c61-b69b3f8a320b" containerID="3d9cdfe263aad4b6fc741769343485d1f64f2b77e754cd96db90e7467c0df063" exitCode=0 Oct 30 07:27:04.919933 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:04.919764 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" event={"ID":"0e446711-8b80-43b4-9c61-b69b3f8a320b","Type":"ContainerDied","Data":"3d9cdfe263aad4b6fc741769343485d1f64f2b77e754cd96db90e7467c0df063"} Oct 30 07:27:04.920308 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:04.920287 4951 scope.go:122] "RemoveContainer" containerID="3d9cdfe263aad4b6fc741769343485d1f64f2b77e754cd96db90e7467c0df063" Oct 30 07:27:04.932181 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:04.921923 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:27:04.932181 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:04.922019 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:27:04.932181 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:04.922064 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:27:05.341598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:05.335329 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 07:27:05.341598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:05.335388 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" podUID="088ac46a-3235-420e-9680-74cb738f229c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 07:27:05.955429 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:05.950860 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" event={"ID":"0e446711-8b80-43b4-9c61-b69b3f8a320b","Type":"ContainerStarted","Data":"66b39227233f23922f618c1ab3d84025fdc4db46f22377d06616e02621293d3d"} Oct 30 07:27:06.340594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:06.337385 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 07:27:06.340594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:06.337443 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" podUID="088ac46a-3235-420e-9680-74cb738f229c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 07:27:06.835448 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:06.830734 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:27:06.835448 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:06.831293 4951 patch_prober.go:29] interesting pod/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 07:27:06.835448 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:06.831375 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" podUID="0e446711-8b80-43b4-9c61-b69b3f8a320b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 07:27:07.342599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:07.335979 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 07:27:07.342599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:07.336041 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" podUID="088ac46a-3235-420e-9680-74cb738f229c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 07:27:07.827067 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:07.827023 4951 patch_prober.go:29] interesting pod/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 07:27:07.827360 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:07.827331 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" podUID="0e446711-8b80-43b4-9c61-b69b3f8a320b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 07:27:08.352597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:08.343236 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 07:27:08.352597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:08.343294 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" podUID="088ac46a-3235-420e-9680-74cb738f229c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 07:27:08.859382 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:08.859341 4951 patch_prober.go:29] interesting pod/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 07:27:08.859641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:08.859404 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" podUID="0e446711-8b80-43b4-9c61-b69b3f8a320b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 07:27:08.991540 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:08.991505 4951 generic.go:356] "Generic (PLEG): container finished" podID="b0938f71-f1d8-4342-8d0a-7aed59adef39" containerID="c30a4cba4d5a0410ce429bd419852e14580de8fe34bc5b07bd2e1a149718e385" exitCode=0 Oct 30 07:27:08.991733 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:08.991593 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" event={"ID":"b0938f71-f1d8-4342-8d0a-7aed59adef39","Type":"ContainerDied","Data":"c30a4cba4d5a0410ce429bd419852e14580de8fe34bc5b07bd2e1a149718e385"} Oct 30 07:27:08.991733 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:08.991679 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" event={"ID":"b0938f71-f1d8-4342-8d0a-7aed59adef39","Type":"ContainerDied","Data":"eaff35d589365094604fa017e4c11dea24d192717ace1e73715b812b7a61718a"} Oct 30 07:27:08.991733 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:08.991703 4951 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eaff35d589365094604fa017e4c11dea24d192717ace1e73715b812b7a61718a" Oct 30 07:27:08.991733 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:08.991734 4951 scope.go:122] "RemoveContainer" containerID="74378f4a114b21f52a0d256fd583040199effa222ac25fd48a58b1e50a660be2" Oct 30 07:27:09.032613 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:09.026835 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:27:09.059768 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:09.059730 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-credential-socket\") pod \"b0938f71-f1d8-4342-8d0a-7aed59adef39\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " Oct 30 07:27:09.059966 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:09.059798 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-istio-envoy\") pod \"b0938f71-f1d8-4342-8d0a-7aed59adef39\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " Oct 30 07:27:09.059966 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:09.059839 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0938f71-f1d8-4342-8d0a-7aed59adef39-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0938f71-f1d8-4342-8d0a-7aed59adef39-istio-podinfo\") pod \"b0938f71-f1d8-4342-8d0a-7aed59adef39\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " Oct 30 07:27:09.060064 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:09.059992 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0938f71-f1d8-4342-8d0a-7aed59adef39-kube-api-access-88s8w\" (UniqueName: \"kubernetes.io/projected/b0938f71-f1d8-4342-8d0a-7aed59adef39-kube-api-access-88s8w\") pod \"b0938f71-f1d8-4342-8d0a-7aed59adef39\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " Oct 30 07:27:09.060064 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:09.060046 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0938f71-f1d8-4342-8d0a-7aed59adef39-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0938f71-f1d8-4342-8d0a-7aed59adef39-istiod-ca-cert\") pod \"b0938f71-f1d8-4342-8d0a-7aed59adef39\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " Oct 30 07:27:09.060144 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:09.060087 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0938f71-f1d8-4342-8d0a-7aed59adef39-istio-token\" (UniqueName: \"kubernetes.io/projected/b0938f71-f1d8-4342-8d0a-7aed59adef39-istio-token\") pod \"b0938f71-f1d8-4342-8d0a-7aed59adef39\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " Oct 30 07:27:09.060144 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:09.060123 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-workload-certs\") pod \"b0938f71-f1d8-4342-8d0a-7aed59adef39\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " Oct 30 07:27:09.060296 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:09.060279 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-istio-data\") pod \"b0938f71-f1d8-4342-8d0a-7aed59adef39\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " Oct 30 07:27:09.060345 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:09.060318 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-workload-socket\") pod \"b0938f71-f1d8-4342-8d0a-7aed59adef39\" (UID: \"b0938f71-f1d8-4342-8d0a-7aed59adef39\") " Oct 30 07:27:09.061241 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:09.061214 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-workload-socket" pod "b0938f71-f1d8-4342-8d0a-7aed59adef39" (UID: "b0938f71-f1d8-4342-8d0a-7aed59adef39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:27:09.061505 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:09.061485 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-credential-socket" pod "b0938f71-f1d8-4342-8d0a-7aed59adef39" (UID: "b0938f71-f1d8-4342-8d0a-7aed59adef39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:27:09.062428 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:09.062182 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-istio-data" pod "b0938f71-f1d8-4342-8d0a-7aed59adef39" (UID: "b0938f71-f1d8-4342-8d0a-7aed59adef39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:27:09.062428 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:09.062229 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0938f71-f1d8-4342-8d0a-7aed59adef39-istiod-ca-cert" pod "b0938f71-f1d8-4342-8d0a-7aed59adef39" (UID: "b0938f71-f1d8-4342-8d0a-7aed59adef39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:27:09.062428 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:09.062393 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-workload-certs" pod "b0938f71-f1d8-4342-8d0a-7aed59adef39" (UID: "b0938f71-f1d8-4342-8d0a-7aed59adef39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:27:09.063952 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:09.063929 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-istio-envoy" pod "b0938f71-f1d8-4342-8d0a-7aed59adef39" (UID: "b0938f71-f1d8-4342-8d0a-7aed59adef39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:27:09.064506 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:09.064484 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0938f71-f1d8-4342-8d0a-7aed59adef39-kube-api-access-88s8w" pod "b0938f71-f1d8-4342-8d0a-7aed59adef39" (UID: "b0938f71-f1d8-4342-8d0a-7aed59adef39"). InnerVolumeSpecName "kube-api-access-88s8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:27:09.064558 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:09.064523 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0938f71-f1d8-4342-8d0a-7aed59adef39-istio-token" pod "b0938f71-f1d8-4342-8d0a-7aed59adef39" (UID: "b0938f71-f1d8-4342-8d0a-7aed59adef39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:27:09.065138 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:09.065119 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0938f71-f1d8-4342-8d0a-7aed59adef39-istio-podinfo" pod "b0938f71-f1d8-4342-8d0a-7aed59adef39" (UID: "b0938f71-f1d8-4342-8d0a-7aed59adef39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:27:09.162353 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:09.162252 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0938f71-f1d8-4342-8d0a-7aed59adef39-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:27:09.162353 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:09.162288 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:27:09.162353 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:09.162308 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:27:09.162353 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:09.162323 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:27:09.162353 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:09.162337 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:27:09.162353 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:09.162354 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0938f71-f1d8-4342-8d0a-7aed59adef39-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:27:09.162809 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:09.162368 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0938f71-f1d8-4342-8d0a-7aed59adef39-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:27:09.162809 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:09.162382 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88s8w\" (UniqueName: \"kubernetes.io/projected/b0938f71-f1d8-4342-8d0a-7aed59adef39-kube-api-access-88s8w\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:27:09.162809 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:09.162396 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0938f71-f1d8-4342-8d0a-7aed59adef39-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:27:09.346458 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:09.345972 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 07:27:09.346458 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:09.346032 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" podUID="088ac46a-3235-420e-9680-74cb738f229c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 07:27:09.827595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:09.826025 4951 patch_prober.go:29] interesting pod/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 07:27:09.827595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:09.826101 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" podUID="0e446711-8b80-43b4-9c61-b69b3f8a320b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 07:27:10.001902 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:10.001872 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv" Oct 30 07:27:10.064763 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:10.064668 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv"] Oct 30 07:27:10.077323 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:10.077287 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-17bd11-openshift-default-74865f446c-wvscv"] Oct 30 07:27:10.800864 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:10.800830 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:27:10.801335 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:10.801215 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:27:10.802488 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:10.802463 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:27:10.825703 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:10.825668 4951 patch_prober.go:29] interesting pod/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 07:27:10.825972 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:10.825934 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" podUID="0e446711-8b80-43b4-9c61-b69b3f8a320b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 07:27:11.773944 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:11.773912 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0938f71-f1d8-4342-8d0a-7aed59adef39" path="/var/lib/kubelet/pods/b0938f71-f1d8-4342-8d0a-7aed59adef39/volumes" Oct 30 07:27:11.826758 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:11.826724 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:27:11.827121 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:11.827103 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:27:11.827766 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:11.827742 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:27:20.511751 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:20.511557 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-59f7e4-c8f484f6c-k9f2h"] Oct 30 07:27:20.641447 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:20.640698 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 07:27:20.664027 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:20.663916 4951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": context deadline exceeded" Oct 30 07:27:20.746600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:20.738990 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:20.772326 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:20.772215 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" podUID="8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.168:8000/ready\": context deadline exceeded" Oct 30 07:27:20.926175 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:20.903441 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:21.105685 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:21.104136 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 07:27:21.155700 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:21.155652 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:21.342718 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:21.342671 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:21.395226 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:21.395115 4951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:22.340826 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:22.340787 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-597d57-7889966d58-vb8fv"] Oct 30 07:27:22.341321 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:22.341027 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-7889966d58-vb8fv" podUID="4152dc51-db44-4f78-b551-3310cf6137c2" containerName="payload-processing" containerID="cri-o://b64094dd5b601f963d574d8f4c0adda84cbc1ef565c298b75731a28c7a8b7e24" gracePeriod=30 Oct 30 07:27:22.375685 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:22.375646 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-866967ccd7-hnk4l"] Oct 30 07:27:22.376216 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:22.376178 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-866967ccd7-hnk4l" podUID="8e0fd956-de2f-4c18-b132-272da9769644" containerName="payload-pre-processing" containerID="cri-o://c6007e510466393655518f1e7915098c4582219714232af61b783dcc09a11cdf" gracePeriod=30 Oct 30 07:27:22.776596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:22.768791 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-7889966d58-vb8fv" Oct 30 07:27:22.849910 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:22.849880 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-866967ccd7-hnk4l" Oct 30 07:27:22.852587 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:22.852552 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4152dc51-db44-4f78-b551-3310cf6137c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4152dc51-db44-4f78-b551-3310cf6137c2-plugins-config-volume\") pod \"4152dc51-db44-4f78-b551-3310cf6137c2\" (UID: \"4152dc51-db44-4f78-b551-3310cf6137c2\") " Oct 30 07:27:22.852726 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:22.852646 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4152dc51-db44-4f78-b551-3310cf6137c2-kube-api-access-57lzq\" (UniqueName: \"kubernetes.io/projected/4152dc51-db44-4f78-b551-3310cf6137c2-kube-api-access-57lzq\") pod \"4152dc51-db44-4f78-b551-3310cf6137c2\" (UID: \"4152dc51-db44-4f78-b551-3310cf6137c2\") " Oct 30 07:27:22.853879 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:22.853854 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4152dc51-db44-4f78-b551-3310cf6137c2-plugins-config-volume" pod "4152dc51-db44-4f78-b551-3310cf6137c2" (UID: "4152dc51-db44-4f78-b551-3310cf6137c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:27:22.855193 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:22.855171 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4152dc51-db44-4f78-b551-3310cf6137c2-kube-api-access-57lzq" pod "4152dc51-db44-4f78-b551-3310cf6137c2" (UID: "4152dc51-db44-4f78-b551-3310cf6137c2"). InnerVolumeSpecName "kube-api-access-57lzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:27:22.953744 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:22.953654 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e0fd956-de2f-4c18-b132-272da9769644-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e0fd956-de2f-4c18-b132-272da9769644-plugins-config-volume\") pod \"8e0fd956-de2f-4c18-b132-272da9769644\" (UID: \"8e0fd956-de2f-4c18-b132-272da9769644\") " Oct 30 07:27:22.953912 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:22.953795 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e0fd956-de2f-4c18-b132-272da9769644-kube-api-access-ww54v\" (UniqueName: \"kubernetes.io/projected/8e0fd956-de2f-4c18-b132-272da9769644-kube-api-access-ww54v\") pod \"8e0fd956-de2f-4c18-b132-272da9769644\" (UID: \"8e0fd956-de2f-4c18-b132-272da9769644\") " Oct 30 07:27:22.954099 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:22.954070 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e0fd956-de2f-4c18-b132-272da9769644-plugins-config-volume" pod "8e0fd956-de2f-4c18-b132-272da9769644" (UID: "8e0fd956-de2f-4c18-b132-272da9769644"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:27:22.954167 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:22.954154 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4152dc51-db44-4f78-b551-3310cf6137c2-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:27:22.954208 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:22.954169 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57lzq\" (UniqueName: \"kubernetes.io/projected/4152dc51-db44-4f78-b551-3310cf6137c2-kube-api-access-57lzq\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:27:22.955977 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:22.955956 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e0fd956-de2f-4c18-b132-272da9769644-kube-api-access-ww54v" pod "8e0fd956-de2f-4c18-b132-272da9769644" (UID: "8e0fd956-de2f-4c18-b132-272da9769644"). InnerVolumeSpecName "kube-api-access-ww54v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:27:23.023794 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:27:23.023753 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:27:23.055954 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:23.055926 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww54v\" (UniqueName: \"kubernetes.io/projected/8e0fd956-de2f-4c18-b132-272da9769644-kube-api-access-ww54v\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:27:23.055954 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:23.055952 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e0fd956-de2f-4c18-b132-272da9769644-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:27:23.204954 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:23.204867 4951 generic.go:356] "Generic (PLEG): container finished" podID="8e0fd956-de2f-4c18-b132-272da9769644" containerID="c6007e510466393655518f1e7915098c4582219714232af61b783dcc09a11cdf" exitCode=0 Oct 30 07:27:23.204954 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:23.204921 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-866967ccd7-hnk4l" Oct 30 07:27:23.205216 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:23.204954 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-866967ccd7-hnk4l" event={"ID":"8e0fd956-de2f-4c18-b132-272da9769644","Type":"ContainerDied","Data":"c6007e510466393655518f1e7915098c4582219714232af61b783dcc09a11cdf"} Oct 30 07:27:23.205216 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:23.205020 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-866967ccd7-hnk4l" event={"ID":"8e0fd956-de2f-4c18-b132-272da9769644","Type":"ContainerDied","Data":"70617db797a9bdea3594aac15e66cbd35caffcb62ab1f2dfb4f44de2c95c1956"} Oct 30 07:27:23.205216 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:23.205063 4951 scope.go:122] "RemoveContainer" containerID="c6007e510466393655518f1e7915098c4582219714232af61b783dcc09a11cdf" Oct 30 07:27:23.207227 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:23.207192 4951 generic.go:356] "Generic (PLEG): container finished" podID="4152dc51-db44-4f78-b551-3310cf6137c2" containerID="b64094dd5b601f963d574d8f4c0adda84cbc1ef565c298b75731a28c7a8b7e24" exitCode=0 Oct 30 07:27:23.207346 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:23.207249 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-7889966d58-vb8fv" Oct 30 07:27:23.207346 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:23.207283 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-7889966d58-vb8fv" event={"ID":"4152dc51-db44-4f78-b551-3310cf6137c2","Type":"ContainerDied","Data":"b64094dd5b601f963d574d8f4c0adda84cbc1ef565c298b75731a28c7a8b7e24"} Oct 30 07:27:23.207346 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:23.207325 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-597d57-7889966d58-vb8fv" event={"ID":"4152dc51-db44-4f78-b551-3310cf6137c2","Type":"ContainerDied","Data":"65b38b921d8c816063bc36d2873f6efb030d1545a226bd501d0acf4546323177"} Oct 30 07:27:23.220226 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:23.220208 4951 scope.go:122] "RemoveContainer" containerID="c6007e510466393655518f1e7915098c4582219714232af61b783dcc09a11cdf" Oct 30 07:27:23.220511 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:27:23.220485 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6007e510466393655518f1e7915098c4582219714232af61b783dcc09a11cdf\": container with ID starting with c6007e510466393655518f1e7915098c4582219714232af61b783dcc09a11cdf not found: ID does not exist" containerID="c6007e510466393655518f1e7915098c4582219714232af61b783dcc09a11cdf" Oct 30 07:27:23.220621 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:23.220525 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6007e510466393655518f1e7915098c4582219714232af61b783dcc09a11cdf"} err="failed to get container status \"c6007e510466393655518f1e7915098c4582219714232af61b783dcc09a11cdf\": rpc error: code = NotFound desc = could not find container \"c6007e510466393655518f1e7915098c4582219714232af61b783dcc09a11cdf\": container with ID starting with c6007e510466393655518f1e7915098c4582219714232af61b783dcc09a11cdf not found: ID does not exist" Oct 30 07:27:23.220621 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:23.220552 4951 scope.go:122] "RemoveContainer" containerID="b64094dd5b601f963d574d8f4c0adda84cbc1ef565c298b75731a28c7a8b7e24" Oct 30 07:27:23.229699 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:23.229684 4951 scope.go:122] "RemoveContainer" containerID="b64094dd5b601f963d574d8f4c0adda84cbc1ef565c298b75731a28c7a8b7e24" Oct 30 07:27:23.229964 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:27:23.229947 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b64094dd5b601f963d574d8f4c0adda84cbc1ef565c298b75731a28c7a8b7e24\": container with ID starting with b64094dd5b601f963d574d8f4c0adda84cbc1ef565c298b75731a28c7a8b7e24 not found: ID does not exist" containerID="b64094dd5b601f963d574d8f4c0adda84cbc1ef565c298b75731a28c7a8b7e24" Oct 30 07:27:23.230015 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:23.229971 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b64094dd5b601f963d574d8f4c0adda84cbc1ef565c298b75731a28c7a8b7e24"} err="failed to get container status \"b64094dd5b601f963d574d8f4c0adda84cbc1ef565c298b75731a28c7a8b7e24\": rpc error: code = NotFound desc = could not find container \"b64094dd5b601f963d574d8f4c0adda84cbc1ef565c298b75731a28c7a8b7e24\": container with ID starting with b64094dd5b601f963d574d8f4c0adda84cbc1ef565c298b75731a28c7a8b7e24 not found: ID does not exist" Oct 30 07:27:23.258344 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:23.258319 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-597d57-7889966d58-vb8fv"] Oct 30 07:27:23.270056 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:23.270023 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-597d57-7889966d58-vb8fv"] Oct 30 07:27:23.301414 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:23.301379 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-866967ccd7-hnk4l"] Oct 30 07:27:23.304528 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:23.304488 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-597d57-866967ccd7-hnk4l"] Oct 30 07:27:23.774329 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:23.774295 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4152dc51-db44-4f78-b551-3310cf6137c2" path="/var/lib/kubelet/pods/4152dc51-db44-4f78-b551-3310cf6137c2/volumes" Oct 30 07:27:23.774795 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:23.774780 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e0fd956-de2f-4c18-b132-272da9769644" path="/var/lib/kubelet/pods/8e0fd956-de2f-4c18-b132-272da9769644/volumes" Oct 30 07:27:25.512801 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:25.512749 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-59f7e4-c8f484f6c-k9f2h" podUID="83322d7f-2879-4260-bc3c-a28fbc7b9b08" containerName="payload-pre-processing" containerID="cri-o://1fa0b74a37564aaf50ca0583a831b02dc224594a60a260819b2306f8250def6f" gracePeriod=25 Oct 30 07:27:25.817824 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:25.817793 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-59f7e4-c8f484f6c-k9f2h" Oct 30 07:27:26.006031 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:26.005997 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83322d7f-2879-4260-bc3c-a28fbc7b9b08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83322d7f-2879-4260-bc3c-a28fbc7b9b08-plugins-config-volume\") pod \"83322d7f-2879-4260-bc3c-a28fbc7b9b08\" (UID: \"83322d7f-2879-4260-bc3c-a28fbc7b9b08\") " Oct 30 07:27:26.006203 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:26.006093 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83322d7f-2879-4260-bc3c-a28fbc7b9b08-kube-api-access-zswdp\" (UniqueName: \"kubernetes.io/projected/83322d7f-2879-4260-bc3c-a28fbc7b9b08-kube-api-access-zswdp\") pod \"83322d7f-2879-4260-bc3c-a28fbc7b9b08\" (UID: \"83322d7f-2879-4260-bc3c-a28fbc7b9b08\") " Oct 30 07:27:26.006375 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:26.006348 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83322d7f-2879-4260-bc3c-a28fbc7b9b08-plugins-config-volume" pod "83322d7f-2879-4260-bc3c-a28fbc7b9b08" (UID: "83322d7f-2879-4260-bc3c-a28fbc7b9b08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:27:26.006674 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:26.006659 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83322d7f-2879-4260-bc3c-a28fbc7b9b08-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:27:26.008261 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:26.008239 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83322d7f-2879-4260-bc3c-a28fbc7b9b08-kube-api-access-zswdp" pod "83322d7f-2879-4260-bc3c-a28fbc7b9b08" (UID: "83322d7f-2879-4260-bc3c-a28fbc7b9b08"). InnerVolumeSpecName "kube-api-access-zswdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:27:26.107598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:26.107486 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zswdp\" (UniqueName: \"kubernetes.io/projected/83322d7f-2879-4260-bc3c-a28fbc7b9b08-kube-api-access-zswdp\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:27:26.230560 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:26.230519 4951 generic.go:356] "Generic (PLEG): container finished" podID="83322d7f-2879-4260-bc3c-a28fbc7b9b08" containerID="1fa0b74a37564aaf50ca0583a831b02dc224594a60a260819b2306f8250def6f" exitCode=0 Oct 30 07:27:26.230560 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:26.230586 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-59f7e4-c8f484f6c-k9f2h" Oct 30 07:27:26.230861 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:26.230613 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-59f7e4-c8f484f6c-k9f2h" event={"ID":"83322d7f-2879-4260-bc3c-a28fbc7b9b08","Type":"ContainerDied","Data":"1fa0b74a37564aaf50ca0583a831b02dc224594a60a260819b2306f8250def6f"} Oct 30 07:27:26.230861 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:26.230688 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-59f7e4-c8f484f6c-k9f2h" event={"ID":"83322d7f-2879-4260-bc3c-a28fbc7b9b08","Type":"ContainerDied","Data":"b687f03cf32f4eda36f28fe38665d46aac97133f19fae46a499e50328ac9d577"} Oct 30 07:27:26.230861 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:26.230717 4951 scope.go:122] "RemoveContainer" containerID="1fa0b74a37564aaf50ca0583a831b02dc224594a60a260819b2306f8250def6f" Oct 30 07:27:26.242818 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:26.242798 4951 scope.go:122] "RemoveContainer" containerID="1fa0b74a37564aaf50ca0583a831b02dc224594a60a260819b2306f8250def6f" Oct 30 07:27:26.243173 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:27:26.243149 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fa0b74a37564aaf50ca0583a831b02dc224594a60a260819b2306f8250def6f\": container with ID starting with 1fa0b74a37564aaf50ca0583a831b02dc224594a60a260819b2306f8250def6f not found: ID does not exist" containerID="1fa0b74a37564aaf50ca0583a831b02dc224594a60a260819b2306f8250def6f" Oct 30 07:27:26.243250 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:26.243183 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fa0b74a37564aaf50ca0583a831b02dc224594a60a260819b2306f8250def6f"} err="failed to get container status \"1fa0b74a37564aaf50ca0583a831b02dc224594a60a260819b2306f8250def6f\": rpc error: code = NotFound desc = could not find container \"1fa0b74a37564aaf50ca0583a831b02dc224594a60a260819b2306f8250def6f\": container with ID starting with 1fa0b74a37564aaf50ca0583a831b02dc224594a60a260819b2306f8250def6f not found: ID does not exist" Oct 30 07:27:26.284537 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:26.284505 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-59f7e4-c8f484f6c-k9f2h"] Oct 30 07:27:26.297949 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:26.297911 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-59f7e4-c8f484f6c-k9f2h"] Oct 30 07:27:27.773500 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:27.773465 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83322d7f-2879-4260-bc3c-a28fbc7b9b08" path="/var/lib/kubelet/pods/83322d7f-2879-4260-bc3c-a28fbc7b9b08/volumes" Oct 30 07:27:30.591906 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:30.591810 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h"] Oct 30 07:27:30.592314 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:30.592103 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" podUID="4f345a13-140a-4122-9ad1-ece45c401b58" containerName="istio-proxy" containerID="cri-o://7b2ccf293ad0ef40186a3fe2ceba17aad67dd3b4aa992d9da406b748293160c7" gracePeriod=30 Oct 30 07:27:32.709613 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.709562 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l"] Oct 30 07:27:32.710481 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.710458 4951 cpu_manager.go:414] "containerMap: removing container" podUID="b0938f71-f1d8-4342-8d0a-7aed59adef39" containerName="istio-proxy" Oct 30 07:27:32.710603 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.710482 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0938f71-f1d8-4342-8d0a-7aed59adef39" containerName="istio-proxy" Oct 30 07:27:32.710603 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.710510 4951 cpu_manager.go:414] "containerMap: removing container" podUID="b0938f71-f1d8-4342-8d0a-7aed59adef39" containerName="istio-proxy" Oct 30 07:27:32.710603 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.710519 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0938f71-f1d8-4342-8d0a-7aed59adef39" containerName="istio-proxy" Oct 30 07:27:32.710603 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.710548 4951 cpu_manager.go:414] "containerMap: removing container" podUID="4152dc51-db44-4f78-b551-3310cf6137c2" containerName="payload-processing" Oct 30 07:27:32.710603 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.710562 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4152dc51-db44-4f78-b551-3310cf6137c2" containerName="payload-processing" Oct 30 07:27:32.710869 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.710613 4951 cpu_manager.go:414] "containerMap: removing container" podUID="8e0fd956-de2f-4c18-b132-272da9769644" containerName="payload-pre-processing" Oct 30 07:27:32.710869 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.710622 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e0fd956-de2f-4c18-b132-272da9769644" containerName="payload-pre-processing" Oct 30 07:27:32.710869 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.710654 4951 cpu_manager.go:414] "containerMap: removing container" podUID="83322d7f-2879-4260-bc3c-a28fbc7b9b08" containerName="payload-pre-processing" Oct 30 07:27:32.710869 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.710661 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83322d7f-2879-4260-bc3c-a28fbc7b9b08" containerName="payload-pre-processing" Oct 30 07:27:32.710869 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.710823 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="4152dc51-db44-4f78-b551-3310cf6137c2" containerName="payload-processing" Oct 30 07:27:32.710869 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.710842 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e0fd956-de2f-4c18-b132-272da9769644" containerName="payload-pre-processing" Oct 30 07:27:32.710869 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.710854 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="83322d7f-2879-4260-bc3c-a28fbc7b9b08" containerName="payload-pre-processing" Oct 30 07:27:32.710869 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.710866 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0938f71-f1d8-4342-8d0a-7aed59adef39" containerName="istio-proxy" Oct 30 07:27:32.711358 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.710878 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0938f71-f1d8-4342-8d0a-7aed59adef39" containerName="istio-proxy" Oct 30 07:27:32.715291 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.715269 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:27:32.758518 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.758493 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-a83dbd-openshift-default-dockercfg-bxlz9\"" Oct 30 07:27:32.762301 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.762274 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l"] Oct 30 07:27:32.830157 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.830120 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istio-envoy\") pod \"e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:27:32.830344 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.830287 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istio-podinfo\") pod \"e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:27:32.830430 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.830408 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-workload-socket\") pod \"e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:27:32.830498 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.830445 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-credential-socket\") pod \"e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:27:32.830638 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.830619 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istiod-ca-cert\") pod \"e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:27:32.830796 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.830774 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-workload-certs\") pod \"e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:27:32.830866 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.830837 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istio-data\") pod \"e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:27:32.830928 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.830872 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8jds\" (UniqueName: \"kubernetes.io/projected/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-kube-api-access-z8jds\") pod \"e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:27:32.831261 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.831235 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istio-token\") pod \"e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:27:32.931996 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.931958 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-workload-socket\") pod \"e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:27:32.931996 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.931992 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-credential-socket\") pod \"e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:27:32.932250 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.932051 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istiod-ca-cert\") pod \"e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:27:32.932250 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.932073 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-workload-certs\") pod \"e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:27:32.932250 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.932103 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istio-data\") pod \"e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:27:32.932250 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.932121 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8jds\" (UniqueName: \"kubernetes.io/projected/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-kube-api-access-z8jds\") pod \"e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:27:32.932250 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.932168 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istio-token\") pod \"e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:27:32.932250 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.932204 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istio-envoy\") pod \"e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:27:32.932605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.932258 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istio-podinfo\") pod \"e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:27:32.932605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.932401 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-workload-socket\") pod \"e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:27:32.932605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.932435 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-workload-certs\") pod \"e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:27:32.932605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.932489 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-credential-socket\") pod \"e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:27:32.932605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.932528 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istio-data\") pod \"e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:27:32.932979 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.932960 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istiod-ca-cert\") pod \"e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:27:32.934683 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.934659 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istio-envoy\") pod \"e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:27:32.934835 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:32.934820 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istio-podinfo\") pod \"e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:27:33.035964 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:33.034340 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8jds\" (UniqueName: \"kubernetes.io/projected/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-kube-api-access-z8jds\") pod \"e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:27:33.036365 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:33.036296 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istio-token\") pod \"e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:27:33.076109 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:33.075727 4951 patch_prober.go:29] interesting pod/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:27:33.076109 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:33.075812 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" podUID="4f345a13-140a-4122-9ad1-ece45c401b58" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:27:33.340649 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:33.340386 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:27:34.773605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:34.756297 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" podUID="8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.168:8000/ready\": context deadline exceeded" Oct 30 07:27:35.078744 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:27:35.078637 4951 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.006s" Oct 30 07:27:35.089601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:35.078985 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l"] Oct 30 07:27:35.089601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:35.079878 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:27:35.089601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:35.079977 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:27:35.089601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:35.080025 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:27:35.195596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:35.195139 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" event={"ID":"43c12ca0-59ed-48ac-8aa7-b72389c8b20d","Type":"ContainerStarted","Data":"8651ce4905cfadcce9e51f09916b10d0cf1e2185731ba641b4b0fa579aefb9f6"} Oct 30 07:27:35.464279 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:35.464219 4951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:35.470977 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:35.470936 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:35.521769 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:35.507700 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:35.568963 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:35.551730 4951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:35.664594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:35.652818 4951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:35.750673 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:27:35.750631 4951 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c12ca0_59ed_48ac_8aa7_b72389c8b20d.slice/crio-8651ce4905cfadcce9e51f09916b10d0cf1e2185731ba641b4b0fa579aefb9f6\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f345a13_140a_4122_9ad1_ece45c401b58.slice/crio-conmon-7b2ccf293ad0ef40186a3fe2ceba17aad67dd3b4aa992d9da406b748293160c7.scope\": RecentStats: unable to find data in memory cache]" Oct 30 07:27:35.751305 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:27:35.751270 4951 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f345a13_140a_4122_9ad1_ece45c401b58.slice/crio-7b2ccf293ad0ef40186a3fe2ceba17aad67dd3b4aa992d9da406b748293160c7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c12ca0_59ed_48ac_8aa7_b72389c8b20d.slice/crio-8651ce4905cfadcce9e51f09916b10d0cf1e2185731ba641b4b0fa579aefb9f6\": RecentStats: unable to find data in memory cache]" Oct 30 07:27:36.044199 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.044166 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:27:36.158804 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.158398 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-credential-socket\") pod \"4f345a13-140a-4122-9ad1-ece45c401b58\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " Oct 30 07:27:36.158804 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.158455 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-istio-data\") pod \"4f345a13-140a-4122-9ad1-ece45c401b58\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " Oct 30 07:27:36.158804 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.158486 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-istio-envoy\") pod \"4f345a13-140a-4122-9ad1-ece45c401b58\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " Oct 30 07:27:36.158804 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.158615 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f345a13-140a-4122-9ad1-ece45c401b58-kube-api-access-kp8xz\" (UniqueName: \"kubernetes.io/projected/4f345a13-140a-4122-9ad1-ece45c401b58-kube-api-access-kp8xz\") pod \"4f345a13-140a-4122-9ad1-ece45c401b58\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " Oct 30 07:27:36.158804 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.158654 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f345a13-140a-4122-9ad1-ece45c401b58-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f345a13-140a-4122-9ad1-ece45c401b58-istiod-ca-cert\") pod \"4f345a13-140a-4122-9ad1-ece45c401b58\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " Oct 30 07:27:36.158804 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.158732 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-workload-certs\") pod \"4f345a13-140a-4122-9ad1-ece45c401b58\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " Oct 30 07:27:36.159241 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.158981 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f345a13-140a-4122-9ad1-ece45c401b58-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f345a13-140a-4122-9ad1-ece45c401b58-istio-podinfo\") pod \"4f345a13-140a-4122-9ad1-ece45c401b58\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " Oct 30 07:27:36.159241 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.159029 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-workload-socket\") pod \"4f345a13-140a-4122-9ad1-ece45c401b58\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " Oct 30 07:27:36.159367 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.159277 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f345a13-140a-4122-9ad1-ece45c401b58-istio-token\" (UniqueName: \"kubernetes.io/projected/4f345a13-140a-4122-9ad1-ece45c401b58-istio-token\") pod \"4f345a13-140a-4122-9ad1-ece45c401b58\" (UID: \"4f345a13-140a-4122-9ad1-ece45c401b58\") " Oct 30 07:27:36.160404 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.160103 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-workload-certs" pod "4f345a13-140a-4122-9ad1-ece45c401b58" (UID: "4f345a13-140a-4122-9ad1-ece45c401b58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:27:36.160798 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.160420 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:27:36.161222 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.161192 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-istio-data" pod "4f345a13-140a-4122-9ad1-ece45c401b58" (UID: "4f345a13-140a-4122-9ad1-ece45c401b58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:27:36.162691 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.162665 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f345a13-140a-4122-9ad1-ece45c401b58-istiod-ca-cert" pod "4f345a13-140a-4122-9ad1-ece45c401b58" (UID: "4f345a13-140a-4122-9ad1-ece45c401b58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:27:36.162798 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.162733 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-credential-socket" pod "4f345a13-140a-4122-9ad1-ece45c401b58" (UID: "4f345a13-140a-4122-9ad1-ece45c401b58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:27:36.163275 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.163242 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-workload-socket" pod "4f345a13-140a-4122-9ad1-ece45c401b58" (UID: "4f345a13-140a-4122-9ad1-ece45c401b58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:27:36.165630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.165604 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f345a13-140a-4122-9ad1-ece45c401b58-istio-token" pod "4f345a13-140a-4122-9ad1-ece45c401b58" (UID: "4f345a13-140a-4122-9ad1-ece45c401b58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:27:36.166379 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.166344 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f345a13-140a-4122-9ad1-ece45c401b58-kube-api-access-kp8xz" pod "4f345a13-140a-4122-9ad1-ece45c401b58" (UID: "4f345a13-140a-4122-9ad1-ece45c401b58"). InnerVolumeSpecName "kube-api-access-kp8xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:27:36.167983 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.167947 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-istio-envoy" pod "4f345a13-140a-4122-9ad1-ece45c401b58" (UID: "4f345a13-140a-4122-9ad1-ece45c401b58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:27:36.174414 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.174387 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f345a13-140a-4122-9ad1-ece45c401b58-istio-podinfo" pod "4f345a13-140a-4122-9ad1-ece45c401b58" (UID: "4f345a13-140a-4122-9ad1-ece45c401b58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:27:36.203643 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.203602 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" event={"ID":"43c12ca0-59ed-48ac-8aa7-b72389c8b20d","Type":"ContainerStarted","Data":"5a01d05c54e68151a209b79f3a527e066560ffd4e8b4da1590cce92e56e84986"} Oct 30 07:27:36.205470 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.205445 4951 generic.go:356] "Generic (PLEG): container finished" podID="4f345a13-140a-4122-9ad1-ece45c401b58" containerID="7b2ccf293ad0ef40186a3fe2ceba17aad67dd3b4aa992d9da406b748293160c7" exitCode=0 Oct 30 07:27:36.205612 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.205500 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" Oct 30 07:27:36.205612 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.205521 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" event={"ID":"4f345a13-140a-4122-9ad1-ece45c401b58","Type":"ContainerDied","Data":"7b2ccf293ad0ef40186a3fe2ceba17aad67dd3b4aa992d9da406b748293160c7"} Oct 30 07:27:36.205612 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.205583 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h" event={"ID":"4f345a13-140a-4122-9ad1-ece45c401b58","Type":"ContainerDied","Data":"c536caabb5f9910cb1ab27d6c912e88a0247129059617f679f4e9ba9cb0b0332"} Oct 30 07:27:36.205612 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.205611 4951 scope.go:122] "RemoveContainer" containerID="7b2ccf293ad0ef40186a3fe2ceba17aad67dd3b4aa992d9da406b748293160c7" Oct 30 07:27:36.216224 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.216204 4951 scope.go:122] "RemoveContainer" containerID="285427ad353b770ac9cbaa25b34fcaae40926d5fa3e5bc9f62d634e61054a8bd" Oct 30 07:27:36.228066 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.227814 4951 scope.go:122] "RemoveContainer" containerID="7b2ccf293ad0ef40186a3fe2ceba17aad67dd3b4aa992d9da406b748293160c7" Oct 30 07:27:36.228234 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:27:36.228149 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b2ccf293ad0ef40186a3fe2ceba17aad67dd3b4aa992d9da406b748293160c7\": container with ID starting with 7b2ccf293ad0ef40186a3fe2ceba17aad67dd3b4aa992d9da406b748293160c7 not found: ID does not exist" containerID="7b2ccf293ad0ef40186a3fe2ceba17aad67dd3b4aa992d9da406b748293160c7" Oct 30 07:27:36.228234 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.228184 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b2ccf293ad0ef40186a3fe2ceba17aad67dd3b4aa992d9da406b748293160c7"} err="failed to get container status \"7b2ccf293ad0ef40186a3fe2ceba17aad67dd3b4aa992d9da406b748293160c7\": rpc error: code = NotFound desc = could not find container \"7b2ccf293ad0ef40186a3fe2ceba17aad67dd3b4aa992d9da406b748293160c7\": container with ID starting with 7b2ccf293ad0ef40186a3fe2ceba17aad67dd3b4aa992d9da406b748293160c7 not found: ID does not exist" Oct 30 07:27:36.228234 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.228214 4951 scope.go:122] "RemoveContainer" containerID="285427ad353b770ac9cbaa25b34fcaae40926d5fa3e5bc9f62d634e61054a8bd" Oct 30 07:27:36.228549 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:27:36.228522 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"285427ad353b770ac9cbaa25b34fcaae40926d5fa3e5bc9f62d634e61054a8bd\": container with ID starting with 285427ad353b770ac9cbaa25b34fcaae40926d5fa3e5bc9f62d634e61054a8bd not found: ID does not exist" containerID="285427ad353b770ac9cbaa25b34fcaae40926d5fa3e5bc9f62d634e61054a8bd" Oct 30 07:27:36.228665 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.228557 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"285427ad353b770ac9cbaa25b34fcaae40926d5fa3e5bc9f62d634e61054a8bd"} err="failed to get container status \"285427ad353b770ac9cbaa25b34fcaae40926d5fa3e5bc9f62d634e61054a8bd\": rpc error: code = NotFound desc = could not find container \"285427ad353b770ac9cbaa25b34fcaae40926d5fa3e5bc9f62d634e61054a8bd\": container with ID starting with 285427ad353b770ac9cbaa25b34fcaae40926d5fa3e5bc9f62d634e61054a8bd not found: ID does not exist" Oct 30 07:27:36.268596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.266063 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:27:36.268596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.266106 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:27:36.268596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.266120 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:27:36.268596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.266132 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp8xz\" (UniqueName: \"kubernetes.io/projected/4f345a13-140a-4122-9ad1-ece45c401b58-kube-api-access-kp8xz\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:27:36.268596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.266148 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f345a13-140a-4122-9ad1-ece45c401b58-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:27:36.268596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.266160 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f345a13-140a-4122-9ad1-ece45c401b58-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:27:36.268596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.266173 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f345a13-140a-4122-9ad1-ece45c401b58-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:27:36.268596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.266200 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f345a13-140a-4122-9ad1-ece45c401b58-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:27:36.347591 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.344867 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:27:36.418688 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.418556 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" podStartSLOduration=4.418535323 podStartE2EDuration="4.418535323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:27:36.363762011 +0000 UTC m=+1862.706809977" watchObservedRunningTime="2026-10-30 07:27:36.418535323 +0000 UTC m=+1862.761583291" Oct 30 07:27:36.484204 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.484162 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h"] Oct 30 07:27:36.489460 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.489433 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:27:36.496002 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:36.494334 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-597d57-openshift-default-56f6598cdc-6nv8h"] Oct 30 07:27:37.213259 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:37.213225 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:27:37.214264 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:37.214247 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:27:37.772680 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:37.772638 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f345a13-140a-4122-9ad1-ece45c401b58" path="/var/lib/kubelet/pods/4f345a13-140a-4122-9ad1-ece45c401b58/volumes" Oct 30 07:27:40.668802 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:40.668762 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a83dbd-66b9d45d75-xr9dv"] Oct 30 07:27:40.669615 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:40.669554 4951 cpu_manager.go:414] "containerMap: removing container" podUID="4f345a13-140a-4122-9ad1-ece45c401b58" containerName="istio-proxy" Oct 30 07:27:40.669615 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:40.669591 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f345a13-140a-4122-9ad1-ece45c401b58" containerName="istio-proxy" Oct 30 07:27:40.669799 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:40.669786 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f345a13-140a-4122-9ad1-ece45c401b58" containerName="istio-proxy" Oct 30 07:27:40.674835 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:40.674816 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a83dbd-66b9d45d75-xr9dv" Oct 30 07:27:40.702320 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:40.702210 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-a83dbd-dockercfg-4tndw\"" Oct 30 07:27:40.702320 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:40.702283 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-a83dbd\"" Oct 30 07:27:40.707028 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:40.706997 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a83dbd-66b9d45d75-xr9dv"] Oct 30 07:27:40.875605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:40.872682 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a83dbd-cb4df464-2rwhk"] Oct 30 07:27:40.875605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:40.873834 4951 cpu_manager.go:414] "containerMap: removing container" podUID="4f345a13-140a-4122-9ad1-ece45c401b58" containerName="istio-proxy" Oct 30 07:27:40.875605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:40.873858 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f345a13-140a-4122-9ad1-ece45c401b58" containerName="istio-proxy" Oct 30 07:27:40.875605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:40.874051 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f345a13-140a-4122-9ad1-ece45c401b58" containerName="istio-proxy" Oct 30 07:27:40.907472 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:40.903119 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a83dbd-cb4df464-2rwhk" Oct 30 07:27:40.932521 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:40.932437 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a83dbd-cb4df464-2rwhk"] Oct 30 07:27:40.936801 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:40.936715 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a96460bf-ba0e-415e-8713-f4da5168996d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a83dbd-66b9d45d75-xr9dv\" (UID: \"a96460bf-ba0e-415e-8713-f4da5168996d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a83dbd-66b9d45d75-xr9dv" Oct 30 07:27:40.937130 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:40.937107 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q924\" (UniqueName: \"kubernetes.io/projected/9d414105-ff0b-4ba9-ad13-a2e2a4c4014e-kube-api-access-5q924\") pod \"payload-processing-e2e-praxis-a-a83dbd-cb4df464-2rwhk\" (UID: \"9d414105-ff0b-4ba9-ad13-a2e2a4c4014e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a83dbd-cb4df464-2rwhk" Oct 30 07:27:40.937330 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:40.937311 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hww2r\" (UniqueName: \"kubernetes.io/projected/a96460bf-ba0e-415e-8713-f4da5168996d-kube-api-access-hww2r\") pod \"payload-pre-processing-e2e-praxis-a-a83dbd-66b9d45d75-xr9dv\" (UID: \"a96460bf-ba0e-415e-8713-f4da5168996d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a83dbd-66b9d45d75-xr9dv" Oct 30 07:27:40.937559 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:40.937541 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d414105-ff0b-4ba9-ad13-a2e2a4c4014e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a83dbd-cb4df464-2rwhk\" (UID: \"9d414105-ff0b-4ba9-ad13-a2e2a4c4014e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a83dbd-cb4df464-2rwhk" Oct 30 07:27:41.038947 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:41.038909 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a96460bf-ba0e-415e-8713-f4da5168996d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a83dbd-66b9d45d75-xr9dv\" (UID: \"a96460bf-ba0e-415e-8713-f4da5168996d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a83dbd-66b9d45d75-xr9dv" Oct 30 07:27:41.039203 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:41.038975 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q924\" (UniqueName: \"kubernetes.io/projected/9d414105-ff0b-4ba9-ad13-a2e2a4c4014e-kube-api-access-5q924\") pod \"payload-processing-e2e-praxis-a-a83dbd-cb4df464-2rwhk\" (UID: \"9d414105-ff0b-4ba9-ad13-a2e2a4c4014e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a83dbd-cb4df464-2rwhk" Oct 30 07:27:41.039203 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:41.039010 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hww2r\" (UniqueName: \"kubernetes.io/projected/a96460bf-ba0e-415e-8713-f4da5168996d-kube-api-access-hww2r\") pod \"payload-pre-processing-e2e-praxis-a-a83dbd-66b9d45d75-xr9dv\" (UID: \"a96460bf-ba0e-415e-8713-f4da5168996d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a83dbd-66b9d45d75-xr9dv" Oct 30 07:27:41.039203 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:41.039081 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d414105-ff0b-4ba9-ad13-a2e2a4c4014e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a83dbd-cb4df464-2rwhk\" (UID: \"9d414105-ff0b-4ba9-ad13-a2e2a4c4014e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a83dbd-cb4df464-2rwhk" Oct 30 07:27:41.039940 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:41.039914 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d414105-ff0b-4ba9-ad13-a2e2a4c4014e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a83dbd-cb4df464-2rwhk\" (UID: \"9d414105-ff0b-4ba9-ad13-a2e2a4c4014e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a83dbd-cb4df464-2rwhk" Oct 30 07:27:41.039940 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:41.039924 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a96460bf-ba0e-415e-8713-f4da5168996d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a83dbd-66b9d45d75-xr9dv\" (UID: \"a96460bf-ba0e-415e-8713-f4da5168996d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a83dbd-66b9d45d75-xr9dv" Oct 30 07:27:41.067412 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:41.063759 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q924\" (UniqueName: \"kubernetes.io/projected/9d414105-ff0b-4ba9-ad13-a2e2a4c4014e-kube-api-access-5q924\") pod \"payload-processing-e2e-praxis-a-a83dbd-cb4df464-2rwhk\" (UID: \"9d414105-ff0b-4ba9-ad13-a2e2a4c4014e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a83dbd-cb4df464-2rwhk" Oct 30 07:27:41.075384 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:41.075353 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hww2r\" (UniqueName: \"kubernetes.io/projected/a96460bf-ba0e-415e-8713-f4da5168996d-kube-api-access-hww2r\") pod \"payload-pre-processing-e2e-praxis-a-a83dbd-66b9d45d75-xr9dv\" (UID: \"a96460bf-ba0e-415e-8713-f4da5168996d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a83dbd-66b9d45d75-xr9dv" Oct 30 07:27:41.268815 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:41.268779 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a83dbd-cb4df464-2rwhk" Oct 30 07:27:41.289872 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:41.289832 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a83dbd-66b9d45d75-xr9dv" Oct 30 07:27:41.582664 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:41.582635 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a83dbd-cb4df464-2rwhk"] Oct 30 07:27:41.583882 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:27:41.583857 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d414105_ff0b_4ba9_ad13_a2e2a4c4014e.slice/crio-7cc1116d3bcd08216b79eb87febaffc695b2feaa7bada319f4d69d5d534bfe7c WatchSource:0}: Error finding container 7cc1116d3bcd08216b79eb87febaffc695b2feaa7bada319f4d69d5d534bfe7c: Status 404 returned error can't find the container with id 7cc1116d3bcd08216b79eb87febaffc695b2feaa7bada319f4d69d5d534bfe7c Oct 30 07:27:41.616279 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:41.616256 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a83dbd-66b9d45d75-xr9dv"] Oct 30 07:27:41.620789 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:27:41.620763 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda96460bf_ba0e_415e_8713_f4da5168996d.slice/crio-5885a1c66a679db7f024395e82a7add32be2bafd86af0c951b754c44e9c73189 WatchSource:0}: Error finding container 5885a1c66a679db7f024395e82a7add32be2bafd86af0c951b754c44e9c73189: Status 404 returned error can't find the container with id 5885a1c66a679db7f024395e82a7add32be2bafd86af0c951b754c44e9c73189 Oct 30 07:27:42.288589 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:42.288106 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a83dbd-cb4df464-2rwhk" event={"ID":"9d414105-ff0b-4ba9-ad13-a2e2a4c4014e","Type":"ContainerStarted","Data":"d6839ba7dc11bda8faecde4afde2bbdc49a24f1cf107f9e1eb0d766fa25bac3c"} Oct 30 07:27:42.288589 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:42.288169 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a83dbd-cb4df464-2rwhk" event={"ID":"9d414105-ff0b-4ba9-ad13-a2e2a4c4014e","Type":"ContainerStarted","Data":"7cc1116d3bcd08216b79eb87febaffc695b2feaa7bada319f4d69d5d534bfe7c"} Oct 30 07:27:42.288589 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:42.288230 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-a83dbd-cb4df464-2rwhk" Oct 30 07:27:42.290154 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:42.290129 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a83dbd-66b9d45d75-xr9dv" event={"ID":"a96460bf-ba0e-415e-8713-f4da5168996d","Type":"ContainerStarted","Data":"1438fd612c0e84b351a67395f2709f516653daef83278d31c3710c940938d3fc"} Oct 30 07:27:42.290302 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:42.290176 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a83dbd-66b9d45d75-xr9dv" event={"ID":"a96460bf-ba0e-415e-8713-f4da5168996d","Type":"ContainerStarted","Data":"5885a1c66a679db7f024395e82a7add32be2bafd86af0c951b754c44e9c73189"} Oct 30 07:27:42.290302 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:42.290203 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a83dbd-66b9d45d75-xr9dv" Oct 30 07:27:42.329342 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:42.329284 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-a83dbd-cb4df464-2rwhk" podStartSLOduration=2.329267299 podStartE2EDuration="2.329267299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:27:42.328436397 +0000 UTC m=+1868.671484364" watchObservedRunningTime="2026-10-30 07:27:42.329267299 +0000 UTC m=+1868.672315264" Oct 30 07:27:42.382485 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:42.382433 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a83dbd-66b9d45d75-xr9dv" podStartSLOduration=2.382418618 podStartE2EDuration="2.382418618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:27:42.380285508 +0000 UTC m=+1868.723333484" watchObservedRunningTime="2026-10-30 07:27:42.382418618 +0000 UTC m=+1868.725466582" Oct 30 07:27:47.108147 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.108105 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp"] Oct 30 07:27:47.112544 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.112522 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:27:47.150370 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.150241 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp"] Oct 30 07:27:47.232139 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.232102 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgzn4\" (UniqueName: \"kubernetes.io/projected/da92d4f9-c801-4fc9-9eaf-589305c6851d-kube-api-access-jgzn4\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:27:47.232341 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.232158 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da92d4f9-c801-4fc9-9eaf-589305c6851d-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:27:47.232341 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.232273 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:27:47.232341 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.232336 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:27:47.232517 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.232372 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:27:47.232517 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.232451 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:27:47.232517 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.232495 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:27:47.232725 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.232520 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da92d4f9-c801-4fc9-9eaf-589305c6851d-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:27:47.232725 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.232617 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da92d4f9-c801-4fc9-9eaf-589305c6851d-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:27:47.335934 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.335895 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgzn4\" (UniqueName: \"kubernetes.io/projected/da92d4f9-c801-4fc9-9eaf-589305c6851d-kube-api-access-jgzn4\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:27:47.338031 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.337999 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da92d4f9-c801-4fc9-9eaf-589305c6851d-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:27:47.338204 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.338082 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:27:47.338204 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.338182 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:27:47.338337 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.338231 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:27:47.338337 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.338286 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:27:47.338436 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.338355 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:27:47.338436 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.338392 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da92d4f9-c801-4fc9-9eaf-589305c6851d-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:27:47.338436 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.338423 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da92d4f9-c801-4fc9-9eaf-589305c6851d-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:27:47.339251 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.339223 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da92d4f9-c801-4fc9-9eaf-589305c6851d-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:27:47.339783 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.339761 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:27:47.340246 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.340169 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:27:47.340592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.340281 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:27:47.340719 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.340525 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:27:47.341396 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.341374 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f4a665-c894f7bc5-ccjfc"] Oct 30 07:27:47.343158 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.343131 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da92d4f9-c801-4fc9-9eaf-589305c6851d-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:27:47.343534 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.343500 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:27:47.362159 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.362076 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da92d4f9-c801-4fc9-9eaf-589305c6851d-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:27:47.365138 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.365116 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgzn4\" (UniqueName: \"kubernetes.io/projected/da92d4f9-c801-4fc9-9eaf-589305c6851d-kube-api-access-jgzn4\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:27:47.428831 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.428782 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:27:47.751960 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.751931 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp"] Oct 30 07:27:47.753260 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:27:47.753236 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda92d4f9_c801_4fc9_9eaf_589305c6851d.slice/crio-29ce253d5cf5127add75b837428b361a21323a392e3fd97d66631e9cb56101ee WatchSource:0}: Error finding container 29ce253d5cf5127add75b837428b361a21323a392e3fd97d66631e9cb56101ee: Status 404 returned error can't find the container with id 29ce253d5cf5127add75b837428b361a21323a392e3fd97d66631e9cb56101ee Oct 30 07:27:47.756273 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.756222 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:27:47.756361 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.756315 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:27:47.756361 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:47.756356 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:27:48.341644 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:48.341601 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" event={"ID":"da92d4f9-c801-4fc9-9eaf-589305c6851d","Type":"ContainerStarted","Data":"1be142c2868170592e4d7287912bc27beb9c90a4acf8158a8fe17496a75d9a83"} Oct 30 07:27:48.342070 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:48.341673 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" event={"ID":"da92d4f9-c801-4fc9-9eaf-589305c6851d","Type":"ContainerStarted","Data":"29ce253d5cf5127add75b837428b361a21323a392e3fd97d66631e9cb56101ee"} Oct 30 07:27:48.388050 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:48.388000 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" podStartSLOduration=1.387986746 podStartE2EDuration="1.387986746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:27:48.386168811 +0000 UTC m=+1874.729216775" watchObservedRunningTime="2026-10-30 07:27:48.387986746 +0000 UTC m=+1874.731034769" Oct 30 07:27:48.429943 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:48.429910 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:27:48.431771 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:48.431738 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 07:27:48.431893 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:48.431798 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" podUID="da92d4f9-c801-4fc9-9eaf-589305c6851d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 07:27:49.444594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:49.432033 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 07:27:49.444594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:49.432097 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" podUID="da92d4f9-c801-4fc9-9eaf-589305c6851d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 07:27:50.440598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:50.438912 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 07:27:50.440598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:50.438974 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" podUID="da92d4f9-c801-4fc9-9eaf-589305c6851d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 07:27:51.437596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:51.434063 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 07:27:51.437596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:51.434147 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" podUID="da92d4f9-c801-4fc9-9eaf-589305c6851d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 07:27:52.156593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:52.154037 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d"] Oct 30 07:27:52.156593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:52.154302 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d" podUID="0149e3ab-1dff-46b7-acef-d557a1bb83e0" containerName="maas-api" containerID="cri-o://db7d72fbbe9443301e77e5e3e264f340ab0c2546b16fad60736501bfeb0f3ac1" gracePeriod=30 Oct 30 07:27:52.348037 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:52.347826 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4a665-c894f7bc5-ccjfc" podUID="9ab4c115-8064-41df-96e2-be5cde2e4846" containerName="payload-pre-processing" containerID="cri-o://e08306491db9bb8603107230bdf350736b06f5f59d686b6e3cf96728855069e8" gracePeriod=25 Oct 30 07:27:52.447665 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:52.446988 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 07:27:52.447665 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:52.447068 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" podUID="da92d4f9-c801-4fc9-9eaf-589305c6851d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 07:27:52.523616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:52.508287 4951 generic.go:356] "Generic (PLEG): container finished" podID="0149e3ab-1dff-46b7-acef-d557a1bb83e0" containerID="db7d72fbbe9443301e77e5e3e264f340ab0c2546b16fad60736501bfeb0f3ac1" exitCode=0 Oct 30 07:27:52.523616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:52.508375 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d" event={"ID":"0149e3ab-1dff-46b7-acef-d557a1bb83e0","Type":"ContainerDied","Data":"db7d72fbbe9443301e77e5e3e264f340ab0c2546b16fad60736501bfeb0f3ac1"} Oct 30 07:27:52.828595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:52.827936 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d" Oct 30 07:27:52.899929 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:52.899899 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb"] Oct 30 07:27:52.900873 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:52.900848 4951 cpu_manager.go:414] "containerMap: removing container" podUID="0149e3ab-1dff-46b7-acef-d557a1bb83e0" containerName="maas-api" Oct 30 07:27:52.901062 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:52.901036 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0149e3ab-1dff-46b7-acef-d557a1bb83e0" containerName="maas-api" Oct 30 07:27:52.901392 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:52.901368 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="0149e3ab-1dff-46b7-acef-d557a1bb83e0" containerName="maas-api" Oct 30 07:27:52.905185 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:52.905161 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb" Oct 30 07:27:52.981659 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:52.971149 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-a83dbd-serving-cert\"" Oct 30 07:27:52.981659 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:52.971695 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0149e3ab-1dff-46b7-acef-d557a1bb83e0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0149e3ab-1dff-46b7-acef-d557a1bb83e0-maas-api-tls\") pod \"0149e3ab-1dff-46b7-acef-d557a1bb83e0\" (UID: \"0149e3ab-1dff-46b7-acef-d557a1bb83e0\") " Oct 30 07:27:52.981902 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:52.981697 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0149e3ab-1dff-46b7-acef-d557a1bb83e0-kube-api-access-ptsdn\" (UniqueName: \"kubernetes.io/projected/0149e3ab-1dff-46b7-acef-d557a1bb83e0-kube-api-access-ptsdn\") pod \"0149e3ab-1dff-46b7-acef-d557a1bb83e0\" (UID: \"0149e3ab-1dff-46b7-acef-d557a1bb83e0\") " Oct 30 07:27:52.981902 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:52.981815 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0149e3ab-1dff-46b7-acef-d557a1bb83e0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0149e3ab-1dff-46b7-acef-d557a1bb83e0-openshift-service-ca\") pod \"0149e3ab-1dff-46b7-acef-d557a1bb83e0\" (UID: \"0149e3ab-1dff-46b7-acef-d557a1bb83e0\") " Oct 30 07:27:52.981902 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:52.981848 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0149e3ab-1dff-46b7-acef-d557a1bb83e0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0149e3ab-1dff-46b7-acef-d557a1bb83e0-metrics-certs\") pod \"0149e3ab-1dff-46b7-acef-d557a1bb83e0\" (UID: \"0149e3ab-1dff-46b7-acef-d557a1bb83e0\") " Oct 30 07:27:52.985980 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:52.985929 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0149e3ab-1dff-46b7-acef-d557a1bb83e0-metrics-certs" pod "0149e3ab-1dff-46b7-acef-d557a1bb83e0" (UID: "0149e3ab-1dff-46b7-acef-d557a1bb83e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:27:52.986314 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:52.986266 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0149e3ab-1dff-46b7-acef-d557a1bb83e0-openshift-service-ca" pod "0149e3ab-1dff-46b7-acef-d557a1bb83e0" (UID: "0149e3ab-1dff-46b7-acef-d557a1bb83e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:27:52.986557 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:52.986531 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0149e3ab-1dff-46b7-acef-d557a1bb83e0-kube-api-access-ptsdn" pod "0149e3ab-1dff-46b7-acef-d557a1bb83e0" (UID: "0149e3ab-1dff-46b7-acef-d557a1bb83e0"). InnerVolumeSpecName "kube-api-access-ptsdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:27:52.987661 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:52.987634 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb"] Oct 30 07:27:52.990710 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:52.990686 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0149e3ab-1dff-46b7-acef-d557a1bb83e0-maas-api-tls" pod "0149e3ab-1dff-46b7-acef-d557a1bb83e0" (UID: "0149e3ab-1dff-46b7-acef-d557a1bb83e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:27:53.018759 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.018491 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4a665-c894f7bc5-ccjfc" Oct 30 07:27:53.054883 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:27:53.054818 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:27:53.084139 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.084101 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f86f9ff-4988-481e-b347-d2314e33fa71-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb\" (UID: \"9f86f9ff-4988-481e-b347-d2314e33fa71\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb" Oct 30 07:27:53.084358 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.084188 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78bn9\" (UniqueName: \"kubernetes.io/projected/9f86f9ff-4988-481e-b347-d2314e33fa71-kube-api-access-78bn9\") pod \"maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb\" (UID: \"9f86f9ff-4988-481e-b347-d2314e33fa71\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb" Oct 30 07:27:53.084358 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.084253 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f86f9ff-4988-481e-b347-d2314e33fa71-metrics-certs\") pod \"maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb\" (UID: \"9f86f9ff-4988-481e-b347-d2314e33fa71\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb" Oct 30 07:27:53.084488 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.084407 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f86f9ff-4988-481e-b347-d2314e33fa71-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb\" (UID: \"9f86f9ff-4988-481e-b347-d2314e33fa71\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb" Oct 30 07:27:53.084637 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.084618 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0149e3ab-1dff-46b7-acef-d557a1bb83e0-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:27:53.084718 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.084639 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptsdn\" (UniqueName: \"kubernetes.io/projected/0149e3ab-1dff-46b7-acef-d557a1bb83e0-kube-api-access-ptsdn\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:27:53.084718 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.084650 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0149e3ab-1dff-46b7-acef-d557a1bb83e0-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:27:53.084718 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.084659 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0149e3ab-1dff-46b7-acef-d557a1bb83e0-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:27:53.189130 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.189079 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ab4c115-8064-41df-96e2-be5cde2e4846-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ab4c115-8064-41df-96e2-be5cde2e4846-plugins-config-volume\") pod \"9ab4c115-8064-41df-96e2-be5cde2e4846\" (UID: \"9ab4c115-8064-41df-96e2-be5cde2e4846\") " Oct 30 07:27:53.189671 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.189645 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ab4c115-8064-41df-96e2-be5cde2e4846-kube-api-access-vw7jc\" (UniqueName: \"kubernetes.io/projected/9ab4c115-8064-41df-96e2-be5cde2e4846-kube-api-access-vw7jc\") pod \"9ab4c115-8064-41df-96e2-be5cde2e4846\" (UID: \"9ab4c115-8064-41df-96e2-be5cde2e4846\") " Oct 30 07:27:53.190173 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.190149 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f86f9ff-4988-481e-b347-d2314e33fa71-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb\" (UID: \"9f86f9ff-4988-481e-b347-d2314e33fa71\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb" Oct 30 07:27:53.190367 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.190349 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78bn9\" (UniqueName: \"kubernetes.io/projected/9f86f9ff-4988-481e-b347-d2314e33fa71-kube-api-access-78bn9\") pod \"maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb\" (UID: \"9f86f9ff-4988-481e-b347-d2314e33fa71\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb" Oct 30 07:27:53.190485 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.190469 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f86f9ff-4988-481e-b347-d2314e33fa71-metrics-certs\") pod \"maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb\" (UID: \"9f86f9ff-4988-481e-b347-d2314e33fa71\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb" Oct 30 07:27:53.190737 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.190721 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f86f9ff-4988-481e-b347-d2314e33fa71-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb\" (UID: \"9f86f9ff-4988-481e-b347-d2314e33fa71\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb" Oct 30 07:27:53.191773 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.191750 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ab4c115-8064-41df-96e2-be5cde2e4846-plugins-config-volume" pod "9ab4c115-8064-41df-96e2-be5cde2e4846" (UID: "9ab4c115-8064-41df-96e2-be5cde2e4846"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:27:53.198292 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.198265 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f86f9ff-4988-481e-b347-d2314e33fa71-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb\" (UID: \"9f86f9ff-4988-481e-b347-d2314e33fa71\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb" Oct 30 07:27:53.202770 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.202734 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ab4c115-8064-41df-96e2-be5cde2e4846-kube-api-access-vw7jc" pod "9ab4c115-8064-41df-96e2-be5cde2e4846" (UID: "9ab4c115-8064-41df-96e2-be5cde2e4846"). InnerVolumeSpecName "kube-api-access-vw7jc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:27:53.217644 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.216942 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f86f9ff-4988-481e-b347-d2314e33fa71-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb\" (UID: \"9f86f9ff-4988-481e-b347-d2314e33fa71\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb" Oct 30 07:27:53.217644 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.217437 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f86f9ff-4988-481e-b347-d2314e33fa71-metrics-certs\") pod \"maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb\" (UID: \"9f86f9ff-4988-481e-b347-d2314e33fa71\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb" Oct 30 07:27:53.260339 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.260266 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78bn9\" (UniqueName: \"kubernetes.io/projected/9f86f9ff-4988-481e-b347-d2314e33fa71-kube-api-access-78bn9\") pod \"maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb\" (UID: \"9f86f9ff-4988-481e-b347-d2314e33fa71\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb" Oct 30 07:27:53.292286 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.292257 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw7jc\" (UniqueName: \"kubernetes.io/projected/9ab4c115-8064-41df-96e2-be5cde2e4846-kube-api-access-vw7jc\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:27:53.292286 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.292284 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ab4c115-8064-41df-96e2-be5cde2e4846-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:27:53.298293 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.298272 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a83dbd-66b9d45d75-xr9dv" Oct 30 07:27:53.298503 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.298484 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-a83dbd-cb4df464-2rwhk" Oct 30 07:27:53.436597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.433000 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 07:27:53.436597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.433137 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" podUID="da92d4f9-c801-4fc9-9eaf-589305c6851d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 07:27:53.525736 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.525691 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4a665-c894f7bc5-ccjfc" event={"ID":"9ab4c115-8064-41df-96e2-be5cde2e4846","Type":"ContainerDied","Data":"e08306491db9bb8603107230bdf350736b06f5f59d686b6e3cf96728855069e8"} Oct 30 07:27:53.526372 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.526350 4951 scope.go:122] "RemoveContainer" containerID="e08306491db9bb8603107230bdf350736b06f5f59d686b6e3cf96728855069e8" Oct 30 07:27:53.526899 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.526873 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4a665-c894f7bc5-ccjfc" Oct 30 07:27:53.527650 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.521797 4951 generic.go:356] "Generic (PLEG): container finished" podID="9ab4c115-8064-41df-96e2-be5cde2e4846" containerID="e08306491db9bb8603107230bdf350736b06f5f59d686b6e3cf96728855069e8" exitCode=0 Oct 30 07:27:53.527794 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.527770 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4a665-c894f7bc5-ccjfc" event={"ID":"9ab4c115-8064-41df-96e2-be5cde2e4846","Type":"ContainerDied","Data":"01570cd99f96a5b02223dfd7d66c66358d83f502bba7c2b3417ea6f57db1bbbe"} Oct 30 07:27:53.545731 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.545700 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d" Oct 30 07:27:53.545981 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.545951 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d" event={"ID":"0149e3ab-1dff-46b7-acef-d557a1bb83e0","Type":"ContainerDied","Data":"bad24772c9b336a53beb4c8e530a5152a51e4f99b95be586070387a8e0bba3ed"} Oct 30 07:27:53.559916 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.559830 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb" Oct 30 07:27:53.570288 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.570267 4951 scope.go:122] "RemoveContainer" containerID="e08306491db9bb8603107230bdf350736b06f5f59d686b6e3cf96728855069e8" Oct 30 07:27:53.571422 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:27:53.570638 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e08306491db9bb8603107230bdf350736b06f5f59d686b6e3cf96728855069e8\": container with ID starting with e08306491db9bb8603107230bdf350736b06f5f59d686b6e3cf96728855069e8 not found: ID does not exist" containerID="e08306491db9bb8603107230bdf350736b06f5f59d686b6e3cf96728855069e8" Oct 30 07:27:53.571422 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.570678 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e08306491db9bb8603107230bdf350736b06f5f59d686b6e3cf96728855069e8"} err="failed to get container status \"e08306491db9bb8603107230bdf350736b06f5f59d686b6e3cf96728855069e8\": rpc error: code = NotFound desc = could not find container \"e08306491db9bb8603107230bdf350736b06f5f59d686b6e3cf96728855069e8\": container with ID starting with e08306491db9bb8603107230bdf350736b06f5f59d686b6e3cf96728855069e8 not found: ID does not exist" Oct 30 07:27:53.571422 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.570702 4951 scope.go:122] "RemoveContainer" containerID="db7d72fbbe9443301e77e5e3e264f340ab0c2546b16fad60736501bfeb0f3ac1" Oct 30 07:27:53.608940 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.608907 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d"] Oct 30 07:27:53.613772 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.613746 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f4a665-59d44bb445-x5w7d"] Oct 30 07:27:53.684130 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.684098 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f4a665-c894f7bc5-ccjfc"] Oct 30 07:27:53.686930 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.686900 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f4a665-c894f7bc5-ccjfc"] Oct 30 07:27:53.787492 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.787416 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0149e3ab-1dff-46b7-acef-d557a1bb83e0" path="/var/lib/kubelet/pods/0149e3ab-1dff-46b7-acef-d557a1bb83e0/volumes" Oct 30 07:27:53.788185 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.788165 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ab4c115-8064-41df-96e2-be5cde2e4846" path="/var/lib/kubelet/pods/9ab4c115-8064-41df-96e2-be5cde2e4846/volumes" Oct 30 07:27:53.907142 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:53.907102 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb"] Oct 30 07:27:54.430233 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:54.430193 4951 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 07:27:54.430442 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:54.430256 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" podUID="da92d4f9-c801-4fc9-9eaf-589305c6851d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 07:27:54.558420 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:54.558364 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb" event={"ID":"9f86f9ff-4988-481e-b347-d2314e33fa71","Type":"ContainerStarted","Data":"a4d58d6933f26705c93ffc75f76aa9fa7542b85e03f22df5c719c3f2c6f43be9"} Oct 30 07:27:54.559033 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:54.558560 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb" event={"ID":"9f86f9ff-4988-481e-b347-d2314e33fa71","Type":"ContainerStarted","Data":"2cd440149cc0ddc1788953a0f6d0936be932dfbbcdcb8a69af187e0fb7959117"} Oct 30 07:27:54.559033 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:54.558631 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb" Oct 30 07:27:54.601228 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:54.601162 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb" podStartSLOduration=2.1301779 podStartE2EDuration="2.601143272s" totalImagesPullingTime="470.965372ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:27:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:27:54.600011252 +0000 UTC m=+1880.943059216" watchObservedRunningTime="2026-10-30 07:27:54.601143272 +0000 UTC m=+1880.944191237" Oct 30 07:27:55.852764 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:55.852733 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:27:55.853354 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:55.853319 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:27:55.854655 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:55.854633 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:27:58.720595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:58.720154 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:58.736698 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:58.736656 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" podUID="8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.168:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:58.876742 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:58.876687 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 07:27:59.049105 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:59.048702 4951 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:27:59.049105 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:59.048795 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" podUID="9ba10610-90eb-43ad-b937-e6c4d4a9dfb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:59.101707 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:59.100688 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:59.138971 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:59.138901 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 07:27:59.628626 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:59.627629 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:59.749836 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:59.749790 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:59.750942 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:59.750913 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:27:59.798701 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:27:59.798668 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:28:00.101874 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:00.101836 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f"] Oct 30 07:28:00.102126 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:00.102098 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" podUID="0e446711-8b80-43b4-9c61-b69b3f8a320b" containerName="istio-proxy" containerID="cri-o://66b39227233f23922f618c1ab3d84025fdc4db46f22377d06616e02621293d3d" gracePeriod=30 Oct 30 07:28:00.577498 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:00.577470 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb" Oct 30 07:28:05.398302 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.398278 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:28:05.450672 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.450633 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-istio-envoy\") pod \"0e446711-8b80-43b4-9c61-b69b3f8a320b\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " Oct 30 07:28:05.450894 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.450708 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e446711-8b80-43b4-9c61-b69b3f8a320b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e446711-8b80-43b4-9c61-b69b3f8a320b-istio-podinfo\") pod \"0e446711-8b80-43b4-9c61-b69b3f8a320b\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " Oct 30 07:28:05.450894 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.450742 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-credential-socket\") pod \"0e446711-8b80-43b4-9c61-b69b3f8a320b\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " Oct 30 07:28:05.450894 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.450788 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e446711-8b80-43b4-9c61-b69b3f8a320b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e446711-8b80-43b4-9c61-b69b3f8a320b-istiod-ca-cert\") pod \"0e446711-8b80-43b4-9c61-b69b3f8a320b\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " Oct 30 07:28:05.450894 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.450834 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-workload-certs\") pod \"0e446711-8b80-43b4-9c61-b69b3f8a320b\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " Oct 30 07:28:05.451196 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.450921 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-workload-socket\") pod \"0e446711-8b80-43b4-9c61-b69b3f8a320b\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " Oct 30 07:28:05.451196 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.450987 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e446711-8b80-43b4-9c61-b69b3f8a320b-istio-token\" (UniqueName: \"kubernetes.io/projected/0e446711-8b80-43b4-9c61-b69b3f8a320b-istio-token\") pod \"0e446711-8b80-43b4-9c61-b69b3f8a320b\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " Oct 30 07:28:05.451305 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.451232 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-credential-socket" pod "0e446711-8b80-43b4-9c61-b69b3f8a320b" (UID: "0e446711-8b80-43b4-9c61-b69b3f8a320b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:28:05.451952 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.451667 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:28:05.452179 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.452132 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e446711-8b80-43b4-9c61-b69b3f8a320b-istiod-ca-cert" pod "0e446711-8b80-43b4-9c61-b69b3f8a320b" (UID: "0e446711-8b80-43b4-9c61-b69b3f8a320b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:28:05.452399 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.452372 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-workload-socket" pod "0e446711-8b80-43b4-9c61-b69b3f8a320b" (UID: "0e446711-8b80-43b4-9c61-b69b3f8a320b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:28:05.452716 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.452686 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-workload-certs" pod "0e446711-8b80-43b4-9c61-b69b3f8a320b" (UID: "0e446711-8b80-43b4-9c61-b69b3f8a320b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:28:05.456208 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.453636 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-istio-envoy" pod "0e446711-8b80-43b4-9c61-b69b3f8a320b" (UID: "0e446711-8b80-43b4-9c61-b69b3f8a320b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:28:05.456208 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.454202 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e446711-8b80-43b4-9c61-b69b3f8a320b-istio-token" pod "0e446711-8b80-43b4-9c61-b69b3f8a320b" (UID: "0e446711-8b80-43b4-9c61-b69b3f8a320b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:28:05.456208 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.454879 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e446711-8b80-43b4-9c61-b69b3f8a320b-istio-podinfo" pod "0e446711-8b80-43b4-9c61-b69b3f8a320b" (UID: "0e446711-8b80-43b4-9c61-b69b3f8a320b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:28:05.552519 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.552485 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e446711-8b80-43b4-9c61-b69b3f8a320b-kube-api-access-4ctd8\" (UniqueName: \"kubernetes.io/projected/0e446711-8b80-43b4-9c61-b69b3f8a320b-kube-api-access-4ctd8\") pod \"0e446711-8b80-43b4-9c61-b69b3f8a320b\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " Oct 30 07:28:05.552738 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.552557 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-istio-data\") pod \"0e446711-8b80-43b4-9c61-b69b3f8a320b\" (UID: \"0e446711-8b80-43b4-9c61-b69b3f8a320b\") " Oct 30 07:28:05.552841 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.552815 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-istio-data" pod "0e446711-8b80-43b4-9c61-b69b3f8a320b" (UID: "0e446711-8b80-43b4-9c61-b69b3f8a320b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:28:05.553095 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.553077 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e446711-8b80-43b4-9c61-b69b3f8a320b-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:28:05.553095 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.553093 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:28:05.553271 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.553102 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:28:05.553271 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.553112 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e446711-8b80-43b4-9c61-b69b3f8a320b-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:28:05.553271 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.553121 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:28:05.553271 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.553129 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e446711-8b80-43b4-9c61-b69b3f8a320b-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:28:05.553271 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.553136 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e446711-8b80-43b4-9c61-b69b3f8a320b-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:28:05.554766 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.554744 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e446711-8b80-43b4-9c61-b69b3f8a320b-kube-api-access-4ctd8" pod "0e446711-8b80-43b4-9c61-b69b3f8a320b" (UID: "0e446711-8b80-43b4-9c61-b69b3f8a320b"). InnerVolumeSpecName "kube-api-access-4ctd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:28:05.654307 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.654217 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ctd8\" (UniqueName: \"kubernetes.io/projected/0e446711-8b80-43b4-9c61-b69b3f8a320b-kube-api-access-4ctd8\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:28:05.839389 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.839352 4951 generic.go:356] "Generic (PLEG): container finished" podID="0e446711-8b80-43b4-9c61-b69b3f8a320b" containerID="66b39227233f23922f618c1ab3d84025fdc4db46f22377d06616e02621293d3d" exitCode=0 Oct 30 07:28:05.839624 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.839417 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" Oct 30 07:28:05.839624 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.839428 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" event={"ID":"0e446711-8b80-43b4-9c61-b69b3f8a320b","Type":"ContainerDied","Data":"66b39227233f23922f618c1ab3d84025fdc4db46f22377d06616e02621293d3d"} Oct 30 07:28:05.839624 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.839495 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f" event={"ID":"0e446711-8b80-43b4-9c61-b69b3f8a320b","Type":"ContainerDied","Data":"ae63c84f1f748a5e786daa8ba55cdefad6d32269a9bf9384e0c538c608c7b529"} Oct 30 07:28:05.839624 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.839527 4951 scope.go:122] "RemoveContainer" containerID="66b39227233f23922f618c1ab3d84025fdc4db46f22377d06616e02621293d3d" Oct 30 07:28:05.851340 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.851316 4951 scope.go:122] "RemoveContainer" containerID="3d9cdfe263aad4b6fc741769343485d1f64f2b77e754cd96db90e7467c0df063" Oct 30 07:28:05.862016 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.861995 4951 scope.go:122] "RemoveContainer" containerID="66b39227233f23922f618c1ab3d84025fdc4db46f22377d06616e02621293d3d" Oct 30 07:28:05.862373 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:28:05.862348 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66b39227233f23922f618c1ab3d84025fdc4db46f22377d06616e02621293d3d\": container with ID starting with 66b39227233f23922f618c1ab3d84025fdc4db46f22377d06616e02621293d3d not found: ID does not exist" containerID="66b39227233f23922f618c1ab3d84025fdc4db46f22377d06616e02621293d3d" Oct 30 07:28:05.862485 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.862384 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b39227233f23922f618c1ab3d84025fdc4db46f22377d06616e02621293d3d"} err="failed to get container status \"66b39227233f23922f618c1ab3d84025fdc4db46f22377d06616e02621293d3d\": rpc error: code = NotFound desc = could not find container \"66b39227233f23922f618c1ab3d84025fdc4db46f22377d06616e02621293d3d\": container with ID starting with 66b39227233f23922f618c1ab3d84025fdc4db46f22377d06616e02621293d3d not found: ID does not exist" Oct 30 07:28:05.862485 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.862411 4951 scope.go:122] "RemoveContainer" containerID="3d9cdfe263aad4b6fc741769343485d1f64f2b77e754cd96db90e7467c0df063" Oct 30 07:28:05.862752 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:28:05.862719 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d9cdfe263aad4b6fc741769343485d1f64f2b77e754cd96db90e7467c0df063\": container with ID starting with 3d9cdfe263aad4b6fc741769343485d1f64f2b77e754cd96db90e7467c0df063 not found: ID does not exist" containerID="3d9cdfe263aad4b6fc741769343485d1f64f2b77e754cd96db90e7467c0df063" Oct 30 07:28:05.862819 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.862759 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d9cdfe263aad4b6fc741769343485d1f64f2b77e754cd96db90e7467c0df063"} err="failed to get container status \"3d9cdfe263aad4b6fc741769343485d1f64f2b77e754cd96db90e7467c0df063\": rpc error: code = NotFound desc = could not find container \"3d9cdfe263aad4b6fc741769343485d1f64f2b77e754cd96db90e7467c0df063\": container with ID starting with 3d9cdfe263aad4b6fc741769343485d1f64f2b77e754cd96db90e7467c0df063 not found: ID does not exist" Oct 30 07:28:05.898101 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.898069 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f"] Oct 30 07:28:05.911606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:05.911521 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-f4a665-openshift-default-5454975f5c-m8l6f"] Oct 30 07:28:07.773558 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:07.773519 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e446711-8b80-43b4-9c61-b69b3f8a320b" path="/var/lib/kubelet/pods/0e446711-8b80-43b4-9c61-b69b3f8a320b/volumes" Oct 30 07:28:11.875017 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:11.874959 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:12.133925 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:12.133738 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 07:28:12.134691 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:12.134642 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 07:28:12.301226 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:12.301184 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 07:28:12.641596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:12.640656 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:12.801801 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:12.801749 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" podUID="8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.168:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:12.831227 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:12.828716 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:12.896017 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:12.893670 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:12.896017 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:12.894294 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:28:13.156949 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:13.156833 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 07:28:13.167724 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:13.159939 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:13.167724 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:13.161471 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:28:13.167724 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:13.161533 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:28:13.332596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:13.314347 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:13.332596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:13.315054 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:28:13.422678 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:13.349209 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:28:13.669605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:13.663656 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 07:28:13.669605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:13.665170 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:28:13.805598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:13.802756 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" podUID="8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.168:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:13.842598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:13.830014 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 07:28:13.931595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:13.922489 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" Oct 30 07:28:13.940734 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:13.940699 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:28:13.968903 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:13.968874 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:28:13.969180 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:13.942809 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:14.107384 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:14.107277 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:28:14.207594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:14.195925 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:28:14.221385 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:14.221348 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" Oct 30 07:28:14.456934 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:14.456828 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:14.695861 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:14.695808 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:14.812607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:14.809694 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:28:14.991595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:14.990678 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 07:28:15.116775 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:15.116664 4951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded" Oct 30 07:28:15.304641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:15.301649 4951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:15.874685 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:15.874650 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8"] Oct 30 07:28:15.875423 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:15.875405 4951 cpu_manager.go:414] "containerMap: removing container" podUID="0e446711-8b80-43b4-9c61-b69b3f8a320b" containerName="istio-proxy" Oct 30 07:28:15.875475 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:15.875425 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e446711-8b80-43b4-9c61-b69b3f8a320b" containerName="istio-proxy" Oct 30 07:28:15.875475 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:15.875444 4951 cpu_manager.go:414] "containerMap: removing container" podUID="0e446711-8b80-43b4-9c61-b69b3f8a320b" containerName="istio-proxy" Oct 30 07:28:15.875475 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:15.875449 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e446711-8b80-43b4-9c61-b69b3f8a320b" containerName="istio-proxy" Oct 30 07:28:15.875586 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:15.875486 4951 cpu_manager.go:414] "containerMap: removing container" podUID="9ab4c115-8064-41df-96e2-be5cde2e4846" containerName="payload-pre-processing" Oct 30 07:28:15.875586 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:15.875491 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ab4c115-8064-41df-96e2-be5cde2e4846" containerName="payload-pre-processing" Oct 30 07:28:15.875656 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:15.875591 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e446711-8b80-43b4-9c61-b69b3f8a320b" containerName="istio-proxy" Oct 30 07:28:15.875656 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:15.875612 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ab4c115-8064-41df-96e2-be5cde2e4846" containerName="payload-pre-processing" Oct 30 07:28:15.881094 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:15.881075 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:28:15.902349 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:15.902315 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8"] Oct 30 07:28:15.911220 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:15.911166 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-2ddf97-openshift-default-dockercfg-m7wfl\"" Oct 30 07:28:15.989681 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:15.989647 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-workload-socket\") pod \"e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:28:15.989882 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:15.989782 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-istio-envoy\") pod \"e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:28:15.989882 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:15.989844 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19bce129-3294-4690-9029-6ddea9c23448-istio-podinfo\") pod \"e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:28:15.989882 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:15.989873 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvmw4\" (UniqueName: \"kubernetes.io/projected/19bce129-3294-4690-9029-6ddea9c23448-kube-api-access-tvmw4\") pod \"e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:28:15.990031 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:15.989949 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-workload-certs\") pod \"e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:28:15.990031 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:15.989986 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-credential-socket\") pod \"e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:28:15.990118 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:15.990085 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19bce129-3294-4690-9029-6ddea9c23448-istio-token\") pod \"e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:28:15.990168 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:15.990137 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19bce129-3294-4690-9029-6ddea9c23448-istiod-ca-cert\") pod \"e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:28:15.990244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:15.990226 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-istio-data\") pod \"e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:28:16.093512 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:16.092144 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19bce129-3294-4690-9029-6ddea9c23448-istio-token\") pod \"e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:28:16.093512 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:16.092233 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19bce129-3294-4690-9029-6ddea9c23448-istiod-ca-cert\") pod \"e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:28:16.093512 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:16.092265 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-istio-data\") pod \"e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:28:16.093512 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:16.092367 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-workload-socket\") pod \"e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:28:16.093512 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:16.092453 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-istio-envoy\") pod \"e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:28:16.093512 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:16.092480 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19bce129-3294-4690-9029-6ddea9c23448-istio-podinfo\") pod \"e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:28:16.093512 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:16.092510 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvmw4\" (UniqueName: \"kubernetes.io/projected/19bce129-3294-4690-9029-6ddea9c23448-kube-api-access-tvmw4\") pod \"e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:28:16.093512 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:16.092557 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-workload-certs\") pod \"e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:28:16.093512 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:16.092625 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-credential-socket\") pod \"e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:28:16.093512 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:16.093070 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-credential-socket\") pod \"e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:28:16.097350 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:16.095558 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19bce129-3294-4690-9029-6ddea9c23448-istiod-ca-cert\") pod \"e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:28:16.107233 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:16.101789 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19bce129-3294-4690-9029-6ddea9c23448-istio-podinfo\") pod \"e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:28:16.107233 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:16.102972 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-workload-socket\") pod \"e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:28:16.134808 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:16.130659 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-istio-data\") pod \"e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:28:16.134808 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:16.130960 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-workload-certs\") pod \"e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:28:16.157620 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:16.157529 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvmw4\" (UniqueName: \"kubernetes.io/projected/19bce129-3294-4690-9029-6ddea9c23448-kube-api-access-tvmw4\") pod \"e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:28:16.158171 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:16.158144 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-istio-envoy\") pod \"e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:28:16.163600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:16.163267 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19bce129-3294-4690-9029-6ddea9c23448-istio-token\") pod \"e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:28:16.197938 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:16.197894 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:28:16.688040 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:28:16.688009 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19bce129_3294_4690_9029_6ddea9c23448.slice/crio-d10f01d89becc026631831c1510008fcf0f1fa9223655410d5d36505e3a4d167 WatchSource:0}: Error finding container d10f01d89becc026631831c1510008fcf0f1fa9223655410d5d36505e3a4d167: Status 404 returned error can't find the container with id d10f01d89becc026631831c1510008fcf0f1fa9223655410d5d36505e3a4d167 Oct 30 07:28:16.688743 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:16.688720 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8"] Oct 30 07:28:16.690415 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:16.690384 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:28:16.690485 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:16.690447 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:28:16.690485 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:16.690478 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:28:16.792503 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:16.792467 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" event={"ID":"19bce129-3294-4690-9029-6ddea9c23448","Type":"ContainerStarted","Data":"d10f01d89becc026631831c1510008fcf0f1fa9223655410d5d36505e3a4d167"} Oct 30 07:28:17.801389 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:17.801354 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" event={"ID":"19bce129-3294-4690-9029-6ddea9c23448","Type":"ContainerStarted","Data":"8587d680a4913102243ec72cd1d22481cb0bf80f83fb2741a6fc8c140ac141c3"} Oct 30 07:28:17.881745 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:17.881685 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" podStartSLOduration=2.881668182 podStartE2EDuration="2.881668182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:28:17.866298002 +0000 UTC m=+1904.209345986" watchObservedRunningTime="2026-10-30 07:28:17.881668182 +0000 UTC m=+1904.224716219" Oct 30 07:28:18.199485 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:18.199389 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:28:18.205428 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:18.205396 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:28:18.807948 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:18.807912 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:28:18.808765 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:18.808744 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:28:19.152045 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.151954 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx"] Oct 30 07:28:19.153033 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.153012 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e446711-8b80-43b4-9c61-b69b3f8a320b" containerName="istio-proxy" Oct 30 07:28:19.157003 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.156983 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:28:19.181740 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.181710 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx"] Oct 30 07:28:19.189648 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.189618 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-fbfb50-openshift-default-dockercfg-857pd\"" Oct 30 07:28:19.295751 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.295711 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-istio-data\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:28:19.295964 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.295764 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6nkd\" (UniqueName: \"kubernetes.io/projected/15a9c484-d57b-4968-beab-6d0cdab0e59c-kube-api-access-g6nkd\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:28:19.295964 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.295806 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-istio-envoy\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:28:19.295964 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.295875 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-workload-certs\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:28:19.295964 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.295905 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-workload-socket\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:28:19.296176 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.296061 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-credential-socket\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:28:19.296176 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.296126 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a9c484-d57b-4968-beab-6d0cdab0e59c-istiod-ca-cert\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:28:19.296242 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.296212 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a9c484-d57b-4968-beab-6d0cdab0e59c-istio-podinfo\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:28:19.296338 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.296325 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15a9c484-d57b-4968-beab-6d0cdab0e59c-istio-token\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:28:19.397929 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.397891 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a9c484-d57b-4968-beab-6d0cdab0e59c-istio-podinfo\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:28:19.398098 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.398039 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15a9c484-d57b-4968-beab-6d0cdab0e59c-istio-token\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:28:19.398233 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.398210 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-istio-data\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:28:19.398326 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.398280 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6nkd\" (UniqueName: \"kubernetes.io/projected/15a9c484-d57b-4968-beab-6d0cdab0e59c-kube-api-access-g6nkd\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:28:19.398326 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.398305 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-istio-envoy\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:28:19.398443 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.398343 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-workload-certs\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:28:19.398500 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.398464 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-workload-socket\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:28:19.398667 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.398648 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-credential-socket\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:28:19.398731 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.398701 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a9c484-d57b-4968-beab-6d0cdab0e59c-istiod-ca-cert\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:28:19.399320 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.399282 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-istio-data\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:28:19.399417 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.399310 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-workload-certs\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:28:19.399417 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.399362 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-workload-socket\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:28:19.399417 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.399403 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-credential-socket\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:28:19.399548 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.399501 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a9c484-d57b-4968-beab-6d0cdab0e59c-istiod-ca-cert\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:28:19.400829 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.400811 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a9c484-d57b-4968-beab-6d0cdab0e59c-istio-podinfo\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:28:19.401175 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.401153 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-istio-envoy\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:28:19.423367 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.423309 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15a9c484-d57b-4968-beab-6d0cdab0e59c-istio-token\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:28:19.423784 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.423761 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6nkd\" (UniqueName: \"kubernetes.io/projected/15a9c484-d57b-4968-beab-6d0cdab0e59c-kube-api-access-g6nkd\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:28:19.476413 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.476372 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:28:19.810845 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:28:19.810812 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a9c484_d57b_4968_beab_6d0cdab0e59c.slice/crio-466bd514f83aaedf224b094fd54ad1c00f86c1494eb8e3e39fd1598439aaf69e WatchSource:0}: Error finding container 466bd514f83aaedf224b094fd54ad1c00f86c1494eb8e3e39fd1598439aaf69e: Status 404 returned error can't find the container with id 466bd514f83aaedf224b094fd54ad1c00f86c1494eb8e3e39fd1598439aaf69e Oct 30 07:28:19.811919 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.811884 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx"] Oct 30 07:28:19.813834 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.813795 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:28:19.813942 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.813875 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:28:19.813942 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:19.813912 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:28:20.854710 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:20.854667 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" event={"ID":"15a9c484-d57b-4968-beab-6d0cdab0e59c","Type":"ContainerStarted","Data":"8a2299442e9188fde16f146403e54cd2b8a282437436111d937435267adc86b7"} Oct 30 07:28:20.855151 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:20.854738 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" event={"ID":"15a9c484-d57b-4968-beab-6d0cdab0e59c","Type":"ContainerStarted","Data":"466bd514f83aaedf224b094fd54ad1c00f86c1494eb8e3e39fd1598439aaf69e"} Oct 30 07:28:20.957597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:20.954653 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" podStartSLOduration=1.95462726 podStartE2EDuration="1.95462726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:28:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:28:20.934088888 +0000 UTC m=+1907.277136854" watchObservedRunningTime="2026-10-30 07:28:20.95462726 +0000 UTC m=+1907.297675226" Oct 30 07:28:21.477088 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:21.477055 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:28:21.478078 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:21.478040 4951 patch_prober.go:29] interesting pod/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 07:28:21.478217 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:21.478085 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" podUID="15a9c484-d57b-4968-beab-6d0cdab0e59c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 07:28:22.477095 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:22.477035 4951 patch_prober.go:29] interesting pod/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 07:28:22.477741 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:22.477109 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" podUID="15a9c484-d57b-4968-beab-6d0cdab0e59c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 07:28:23.054627 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:28:23.053732 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:28:23.257594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:23.257069 4951 patch_prober.go:29] interesting pod/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:28:23.257594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:23.257139 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" podUID="43c12ca0-59ed-48ac-8aa7-b72389c8b20d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:23.478168 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:23.478087 4951 patch_prober.go:29] interesting pod/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 07:28:23.478168 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:23.478148 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" podUID="15a9c484-d57b-4968-beab-6d0cdab0e59c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 07:28:23.919419 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:23.919388 4951 generic.go:356] "Generic (PLEG): container finished" podID="43c12ca0-59ed-48ac-8aa7-b72389c8b20d" containerID="5a01d05c54e68151a209b79f3a527e066560ffd4e8b4da1590cce92e56e84986" exitCode=0 Oct 30 07:28:23.919630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:23.919453 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" event={"ID":"43c12ca0-59ed-48ac-8aa7-b72389c8b20d","Type":"ContainerDied","Data":"5a01d05c54e68151a209b79f3a527e066560ffd4e8b4da1590cce92e56e84986"} Oct 30 07:28:23.919918 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:23.919899 4951 scope.go:122] "RemoveContainer" containerID="5a01d05c54e68151a209b79f3a527e066560ffd4e8b4da1590cce92e56e84986" Oct 30 07:28:23.921377 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:23.921348 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:28:23.921441 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:23.921420 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:28:23.921486 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:23.921453 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:28:24.477269 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:24.477227 4951 patch_prober.go:29] interesting pod/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 07:28:24.477485 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:24.477277 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" podUID="15a9c484-d57b-4968-beab-6d0cdab0e59c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 07:28:25.084615 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:25.084448 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" event={"ID":"43c12ca0-59ed-48ac-8aa7-b72389c8b20d","Type":"ContainerStarted","Data":"b0f642baf163be09c161669f1b2d24c48f583c66828ad4a059c8ed7689295f19"} Oct 30 07:28:25.347653 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:25.347540 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:28:25.347828 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:25.347683 4951 patch_prober.go:29] interesting pod/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 30 07:28:25.347828 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:25.347724 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" podUID="43c12ca0-59ed-48ac-8aa7-b72389c8b20d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 30 07:28:25.482601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:25.478060 4951 patch_prober.go:29] interesting pod/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 07:28:25.482601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:25.478142 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" podUID="15a9c484-d57b-4968-beab-6d0cdab0e59c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 07:28:26.352780 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:26.352701 4951 patch_prober.go:29] interesting pod/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 30 07:28:26.352780 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:26.352771 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" podUID="43c12ca0-59ed-48ac-8aa7-b72389c8b20d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 30 07:28:26.597967 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:26.597932 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:28:26.598831 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:26.598806 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:28:26.599915 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:26.599889 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:28:27.342596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:27.341958 4951 patch_prober.go:29] interesting pod/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 30 07:28:27.342596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:27.342021 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" podUID="43c12ca0-59ed-48ac-8aa7-b72389c8b20d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 30 07:28:28.347970 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:28.347924 4951 patch_prober.go:29] interesting pod/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 30 07:28:28.348474 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:28.347983 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" podUID="43c12ca0-59ed-48ac-8aa7-b72389c8b20d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 30 07:28:29.372593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:29.361895 4951 patch_prober.go:29] interesting pod/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 30 07:28:29.372593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:29.362013 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" podUID="43c12ca0-59ed-48ac-8aa7-b72389c8b20d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 30 07:28:30.346052 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:30.346022 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:28:30.346434 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:30.346411 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:28:30.347196 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:30.347175 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:28:35.602954 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:35.602922 4951 scope.go:122] "RemoveContainer" containerID="c30a4cba4d5a0410ce429bd419852e14580de8fe34bc5b07bd2e1a149718e385" Oct 30 07:28:43.664218 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:43.664182 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm"] Oct 30 07:28:53.023090 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:28:53.023050 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:28:54.187859 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:54.187825 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg"] Oct 30 07:28:54.193776 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:54.193754 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg" Oct 30 07:28:54.215852 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:54.215813 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg"] Oct 30 07:28:54.242917 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:54.242882 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-2e8b2a-serving-cert\"" Oct 30 07:28:54.346391 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:54.346353 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75cf7bf0-7eb7-4f6c-a5a1-c9436acec579-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg\" (UID: \"75cf7bf0-7eb7-4f6c-a5a1-c9436acec579\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg" Oct 30 07:28:54.346586 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:54.346511 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhlxl\" (UniqueName: \"kubernetes.io/projected/75cf7bf0-7eb7-4f6c-a5a1-c9436acec579-kube-api-access-vhlxl\") pod \"maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg\" (UID: \"75cf7bf0-7eb7-4f6c-a5a1-c9436acec579\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg" Oct 30 07:28:54.346586 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:54.346555 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75cf7bf0-7eb7-4f6c-a5a1-c9436acec579-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg\" (UID: \"75cf7bf0-7eb7-4f6c-a5a1-c9436acec579\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg" Oct 30 07:28:54.346708 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:54.346656 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75cf7bf0-7eb7-4f6c-a5a1-c9436acec579-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg\" (UID: \"75cf7bf0-7eb7-4f6c-a5a1-c9436acec579\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg" Oct 30 07:28:54.449648 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:54.448737 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75cf7bf0-7eb7-4f6c-a5a1-c9436acec579-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg\" (UID: \"75cf7bf0-7eb7-4f6c-a5a1-c9436acec579\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg" Oct 30 07:28:54.449648 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:54.449027 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhlxl\" (UniqueName: \"kubernetes.io/projected/75cf7bf0-7eb7-4f6c-a5a1-c9436acec579-kube-api-access-vhlxl\") pod \"maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg\" (UID: \"75cf7bf0-7eb7-4f6c-a5a1-c9436acec579\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg" Oct 30 07:28:54.449648 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:54.449099 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75cf7bf0-7eb7-4f6c-a5a1-c9436acec579-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg\" (UID: \"75cf7bf0-7eb7-4f6c-a5a1-c9436acec579\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg" Oct 30 07:28:54.449648 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:54.449133 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75cf7bf0-7eb7-4f6c-a5a1-c9436acec579-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg\" (UID: \"75cf7bf0-7eb7-4f6c-a5a1-c9436acec579\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg" Oct 30 07:28:54.463546 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:54.463503 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75cf7bf0-7eb7-4f6c-a5a1-c9436acec579-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg\" (UID: \"75cf7bf0-7eb7-4f6c-a5a1-c9436acec579\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg" Oct 30 07:28:54.464023 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:54.463997 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75cf7bf0-7eb7-4f6c-a5a1-c9436acec579-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg\" (UID: \"75cf7bf0-7eb7-4f6c-a5a1-c9436acec579\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg" Oct 30 07:28:54.478595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:54.477448 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75cf7bf0-7eb7-4f6c-a5a1-c9436acec579-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg\" (UID: \"75cf7bf0-7eb7-4f6c-a5a1-c9436acec579\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg" Oct 30 07:28:54.483255 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:54.483220 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhlxl\" (UniqueName: \"kubernetes.io/projected/75cf7bf0-7eb7-4f6c-a5a1-c9436acec579-kube-api-access-vhlxl\") pod \"maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg\" (UID: \"75cf7bf0-7eb7-4f6c-a5a1-c9436acec579\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg" Oct 30 07:28:54.507810 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:54.507784 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg" Oct 30 07:28:54.816215 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:28:54.816175 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75cf7bf0_7eb7_4f6c_a5a1_c9436acec579.slice/crio-637326e4a89863abdcba84f3ade3c20e40d08e1393bab2bf80d3f64777343cbd WatchSource:0}: Error finding container 637326e4a89863abdcba84f3ade3c20e40d08e1393bab2bf80d3f64777343cbd: Status 404 returned error can't find the container with id 637326e4a89863abdcba84f3ade3c20e40d08e1393bab2bf80d3f64777343cbd Oct 30 07:28:54.817228 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:54.817198 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg"] Oct 30 07:28:55.118448 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.118358 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54"] Oct 30 07:28:55.139541 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.139504 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:28:55.155764 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.155733 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4"] Oct 30 07:28:55.165945 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.165795 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-989104-openshift-default-dockercfg-59vg7\"" Oct 30 07:28:55.166375 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.166351 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54"] Oct 30 07:28:55.166513 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.166467 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:28:55.168032 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.167520 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4"] Oct 30 07:28:55.261258 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.261224 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:28:55.261888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.261297 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:28:55.261888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.261346 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:28:55.261888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.261379 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:28:55.261888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.261430 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ea0e265-4a3b-46ff-98d1-129054c698df-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:28:55.261888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.261478 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:28:55.261888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.261544 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:28:55.261888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.261647 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ea0e265-4a3b-46ff-98d1-129054c698df-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:28:55.261888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.261679 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ea0e265-4a3b-46ff-98d1-129054c698df-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:28:55.261888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.261716 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:28:55.261888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.261750 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:28:55.261888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.261814 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kj98\" (UniqueName: \"kubernetes.io/projected/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-kube-api-access-6kj98\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:28:55.261888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.261856 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:28:55.262483 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.261925 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:28:55.262483 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.261971 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w22j7\" (UniqueName: \"kubernetes.io/projected/2ea0e265-4a3b-46ff-98d1-129054c698df-kube-api-access-w22j7\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:28:55.262483 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.262040 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:28:55.262483 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.262102 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:28:55.262483 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.262161 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:28:55.363996 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.363953 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:28:55.364207 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.364024 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w22j7\" (UniqueName: \"kubernetes.io/projected/2ea0e265-4a3b-46ff-98d1-129054c698df-kube-api-access-w22j7\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:28:55.364207 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.364086 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:28:55.364207 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.364139 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:28:55.364207 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.364174 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:28:55.364207 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.364204 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:28:55.364450 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.364247 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:28:55.364450 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.364288 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:28:55.364450 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.364319 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:28:55.364450 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.364367 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ea0e265-4a3b-46ff-98d1-129054c698df-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:28:55.364450 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.364412 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:28:55.364728 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.364478 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:28:55.366312 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.364957 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ea0e265-4a3b-46ff-98d1-129054c698df-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:28:55.366312 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.365002 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ea0e265-4a3b-46ff-98d1-129054c698df-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:28:55.366312 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.365033 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:28:55.366312 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.365033 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:28:55.366312 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.365077 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:28:55.366312 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.365139 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kj98\" (UniqueName: \"kubernetes.io/projected/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-kube-api-access-6kj98\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:28:55.366312 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.365182 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:28:55.366312 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.365293 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:28:55.366312 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.365358 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:28:55.366312 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.365517 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:28:55.366312 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.365833 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:28:55.366312 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.366008 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ea0e265-4a3b-46ff-98d1-129054c698df-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:28:55.366312 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.366049 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:28:55.366962 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.366849 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:28:55.368180 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.367815 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:28:55.368180 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.367845 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:28:55.368738 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.368394 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:28:55.368738 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.368615 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:28:55.369062 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.369029 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:28:55.369923 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.369897 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ea0e265-4a3b-46ff-98d1-129054c698df-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:28:55.389058 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.389023 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ea0e265-4a3b-46ff-98d1-129054c698df-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:28:55.389250 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.389228 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:28:55.389356 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.389338 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w22j7\" (UniqueName: \"kubernetes.io/projected/2ea0e265-4a3b-46ff-98d1-129054c698df-kube-api-access-w22j7\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:28:55.390253 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.390231 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kj98\" (UniqueName: \"kubernetes.io/projected/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-kube-api-access-6kj98\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:28:55.506877 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.468554 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:28:55.520465 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.520245 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg" event={"ID":"75cf7bf0-7eb7-4f6c-a5a1-c9436acec579","Type":"ContainerStarted","Data":"ffd6e250c5965c0eff17c2781ae999596227151ff676c190a7da11663c7d51fe"} Oct 30 07:28:55.520771 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.520412 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg" event={"ID":"75cf7bf0-7eb7-4f6c-a5a1-c9436acec579","Type":"ContainerStarted","Data":"637326e4a89863abdcba84f3ade3c20e40d08e1393bab2bf80d3f64777343cbd"} Oct 30 07:28:55.521053 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.520841 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:28:55.521261 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.521108 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg" Oct 30 07:28:55.585802 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.584204 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg" podStartSLOduration=1.220665486 podStartE2EDuration="1.584183973s" totalImagesPullingTime="363.518487ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:28:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:28:55.58109319 +0000 UTC m=+1941.924141155" watchObservedRunningTime="2026-10-30 07:28:55.584183973 +0000 UTC m=+1941.927231936" Oct 30 07:28:55.896658 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.894634 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4"] Oct 30 07:28:55.906156 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.906112 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54"] Oct 30 07:28:55.909878 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:28:55.909848 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd13b861_9db1_4e94_bade_fa1a2ca4b04d.slice/crio-170b624da3cf93e62f85c6f4bbd1bb5f4ffaf42a0d5475e94335bd5e13440685 WatchSource:0}: Error finding container 170b624da3cf93e62f85c6f4bbd1bb5f4ffaf42a0d5475e94335bd5e13440685: Status 404 returned error can't find the container with id 170b624da3cf93e62f85c6f4bbd1bb5f4ffaf42a0d5475e94335bd5e13440685 Oct 30 07:28:55.910668 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:28:55.910639 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ea0e265_4a3b_46ff_98d1_129054c698df.slice/crio-d10aa56fc6f0de22ebc4d8ff8dfcc854b2e5305fa071123a6b3e1cb53df612f4 WatchSource:0}: Error finding container d10aa56fc6f0de22ebc4d8ff8dfcc854b2e5305fa071123a6b3e1cb53df612f4: Status 404 returned error can't find the container with id d10aa56fc6f0de22ebc4d8ff8dfcc854b2e5305fa071123a6b3e1cb53df612f4 Oct 30 07:28:55.912653 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.912623 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:28:55.912723 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.912694 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:28:55.912771 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.912726 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:28:55.913256 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.913225 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:28:55.913327 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.913294 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:28:55.913386 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:55.913326 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:28:56.529724 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:56.529680 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" event={"ID":"2ea0e265-4a3b-46ff-98d1-129054c698df","Type":"ContainerStarted","Data":"a75ce083500fd2e5fa40312feaef96567507cceab1f54a5781c4db842454b370"} Oct 30 07:28:56.530188 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:56.529764 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" event={"ID":"2ea0e265-4a3b-46ff-98d1-129054c698df","Type":"ContainerStarted","Data":"d10aa56fc6f0de22ebc4d8ff8dfcc854b2e5305fa071123a6b3e1cb53df612f4"} Oct 30 07:28:56.531982 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:56.531953 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" event={"ID":"bd13b861-9db1-4e94-bade-fa1a2ca4b04d","Type":"ContainerStarted","Data":"7248cda4ade74431263bfdaccff9879761c779383ca885e91c8e80e391c31c7f"} Oct 30 07:28:56.532201 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:56.532005 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" event={"ID":"bd13b861-9db1-4e94-bade-fa1a2ca4b04d","Type":"ContainerStarted","Data":"170b624da3cf93e62f85c6f4bbd1bb5f4ffaf42a0d5475e94335bd5e13440685"} Oct 30 07:28:56.583417 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:56.583366 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" podStartSLOduration=1.583352654 podStartE2EDuration="1.583352654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:28:56.581145815 +0000 UTC m=+1942.924193777" watchObservedRunningTime="2026-10-30 07:28:56.583352654 +0000 UTC m=+1942.926400664" Oct 30 07:28:56.630751 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:56.630702 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" podStartSLOduration=1.6306879840000001 podStartE2EDuration="1.630687984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:28:56.627768501 +0000 UTC m=+1942.970816491" watchObservedRunningTime="2026-10-30 07:28:56.630687984 +0000 UTC m=+1942.973735943" Oct 30 07:28:57.481220 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:57.478828 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:28:57.481613 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:57.481592 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 07:28:57.481829 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:57.481803 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" podUID="2ea0e265-4a3b-46ff-98d1-129054c698df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 07:28:57.528100 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:57.528054 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 07:28:57.528303 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:57.528125 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" podUID="bd13b861-9db1-4e94-bade-fa1a2ca4b04d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 07:28:57.528682 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:57.528659 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:28:58.472209 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:58.472163 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 07:28:58.472922 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:58.472885 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" podUID="2ea0e265-4a3b-46ff-98d1-129054c698df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 07:28:58.541830 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:58.541795 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 07:28:58.542110 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:58.542082 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" podUID="bd13b861-9db1-4e94-bade-fa1a2ca4b04d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 07:28:58.689395 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:58.688929 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" podUID="8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b" containerName="main" containerID="cri-o://197a4e19929d6a1c2a6d8ac3edb7fc2f27865d1b710184561f2a7529311253a6" gracePeriod=45 Oct 30 07:28:59.420594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:59.418949 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" Oct 30 07:28:59.498040 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:59.498003 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 07:28:59.499713 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:59.499675 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" podUID="2ea0e265-4a3b-46ff-98d1-129054c698df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 07:28:59.529642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:59.525667 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 07:28:59.529642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:59.525720 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" podUID="bd13b861-9db1-4e94-bade-fa1a2ca4b04d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 07:28:59.544606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:59.543839 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-home\" (UniqueName: \"kubernetes.io/empty-dir/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-home\") pod \"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b\" (UID: \"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b\") " Oct 30 07:28:59.544606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:59.544003 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-tls-certs\" (UniqueName: \"kubernetes.io/secret/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-tls-certs\") pod \"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b\" (UID: \"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b\") " Oct 30 07:28:59.544606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:59.544089 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-model-cache\") pod \"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b\" (UID: \"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b\") " Oct 30 07:28:59.544606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:59.544126 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-dshm\") pod \"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b\" (UID: \"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b\") " Oct 30 07:28:59.544606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:59.544172 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-kube-api-access-4m8hd\" (UniqueName: \"kubernetes.io/projected/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-kube-api-access-4m8hd\") pod \"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b\" (UID: \"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b\") " Oct 30 07:28:59.544606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:59.544215 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-tmp-dir\") pod \"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b\" (UID: \"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b\") " Oct 30 07:28:59.563268 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:59.545070 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-tmp-dir" pod "8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b" (UID: "8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:28:59.563268 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:59.545267 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-home" pod "8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b" (UID: "8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:28:59.596867 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:59.596828 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-dshm" pod "8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b" (UID: "8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:28:59.601400 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:59.601367 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-model-cache" pod "8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b" (UID: "8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:28:59.604469 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:59.604426 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-kube-api-access-4m8hd" pod "8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b" (UID: "8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b"). InnerVolumeSpecName "kube-api-access-4m8hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:28:59.607385 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:59.607354 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-tls-certs" pod "8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b" (UID: "8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:28:59.612474 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:59.612442 4951 generic.go:356] "Generic (PLEG): container finished" podID="8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b" containerID="197a4e19929d6a1c2a6d8ac3edb7fc2f27865d1b710184561f2a7529311253a6" exitCode=0 Oct 30 07:28:59.612653 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:59.612631 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" event={"ID":"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b","Type":"ContainerDied","Data":"197a4e19929d6a1c2a6d8ac3edb7fc2f27865d1b710184561f2a7529311253a6"} Oct 30 07:28:59.612754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:59.612674 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" event={"ID":"8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b","Type":"ContainerDied","Data":"4b748fd791e647be42c63b2921fcf80e729648c0c7b27024da9f2bbec1c7935d"} Oct 30 07:28:59.612754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:59.612696 4951 scope.go:122] "RemoveContainer" containerID="197a4e19929d6a1c2a6d8ac3edb7fc2f27865d1b710184561f2a7529311253a6" Oct 30 07:28:59.612898 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:59.612875 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" Oct 30 07:28:59.657644 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:59.655233 4951 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-tls-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:28:59.657644 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:59.655267 4951 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-model-cache\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:28:59.657644 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:59.655279 4951 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-dshm\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:28:59.657644 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:59.655291 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m8hd\" (UniqueName: \"kubernetes.io/projected/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-kube-api-access-4m8hd\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:28:59.657644 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:59.655305 4951 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-tmp-dir\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:28:59.657644 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:59.655318 4951 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b-home\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:28:59.708836 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:59.708799 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm"] Oct 30 07:28:59.711560 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:59.711533 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm"] Oct 30 07:28:59.721705 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:59.721515 4951 scope.go:122] "RemoveContainer" containerID="197a4e19929d6a1c2a6d8ac3edb7fc2f27865d1b710184561f2a7529311253a6" Oct 30 07:28:59.722030 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:28:59.721999 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"197a4e19929d6a1c2a6d8ac3edb7fc2f27865d1b710184561f2a7529311253a6\": container with ID starting with 197a4e19929d6a1c2a6d8ac3edb7fc2f27865d1b710184561f2a7529311253a6 not found: ID does not exist" containerID="197a4e19929d6a1c2a6d8ac3edb7fc2f27865d1b710184561f2a7529311253a6" Oct 30 07:28:59.722232 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:59.722203 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"197a4e19929d6a1c2a6d8ac3edb7fc2f27865d1b710184561f2a7529311253a6"} err="failed to get container status \"197a4e19929d6a1c2a6d8ac3edb7fc2f27865d1b710184561f2a7529311253a6\": rpc error: code = NotFound desc = could not find container \"197a4e19929d6a1c2a6d8ac3edb7fc2f27865d1b710184561f2a7529311253a6\": container with ID starting with 197a4e19929d6a1c2a6d8ac3edb7fc2f27865d1b710184561f2a7529311253a6 not found: ID does not exist" Oct 30 07:28:59.796084 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:28:59.796004 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b" path="/var/lib/kubelet/pods/8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b/volumes" Oct 30 07:29:00.254674 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:00.250632 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-fbfb50/praxis-route-fbfb50-kserve-748968b77f-rhkkm" podUID="8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.168:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:29:00.470592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:00.470086 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 07:29:00.470592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:00.470147 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" podUID="2ea0e265-4a3b-46ff-98d1-129054c698df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 07:29:00.526653 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:00.525623 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 07:29:00.526653 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:00.525695 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" podUID="bd13b861-9db1-4e94-bade-fa1a2ca4b04d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 07:29:01.544597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:01.502807 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 07:29:01.545294 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:01.545251 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" podUID="2ea0e265-4a3b-46ff-98d1-129054c698df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 07:29:01.545952 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:01.527019 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 07:29:01.546204 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:01.546173 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" podUID="bd13b861-9db1-4e94-bade-fa1a2ca4b04d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 07:29:01.621170 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:01.621042 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg" Oct 30 07:29:02.058598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:02.035079 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 07:29:02.078119 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:02.066712 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 07:29:02.139763 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:02.139699 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:29:02.402845 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:02.402746 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 07:29:02.470860 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:02.470820 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 07:29:02.471160 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:02.471124 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" podUID="2ea0e265-4a3b-46ff-98d1-129054c698df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 07:29:02.525663 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:02.525625 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 07:29:02.525934 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:02.525908 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" podUID="bd13b861-9db1-4e94-bade-fa1a2ca4b04d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 07:29:03.083610 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:03.081668 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 07:29:03.083610 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:03.082116 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 07:29:03.084253 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:03.084225 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:29:03.084750 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:03.084727 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:29:03.102692 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:03.102648 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 07:29:03.106066 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:03.106038 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:29:03.230048 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:03.229990 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 07:29:03.232450 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:03.231356 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:29:03.488318 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:03.464840 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:29:03.488318 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:03.465377 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:29:03.488318 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:03.485268 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 07:29:03.488318 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:03.485345 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" podUID="2ea0e265-4a3b-46ff-98d1-129054c698df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 07:29:03.563777 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:03.563728 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 07:29:03.564052 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:03.564022 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" podUID="bd13b861-9db1-4e94-bade-fa1a2ca4b04d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 07:29:04.083996 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:04.083952 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:29:04.158597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:04.146770 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:29:04.198829 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:04.162612 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:29:04.199823 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:04.199784 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 07:29:04.233991 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:04.233916 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 07:29:04.469650 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:04.466855 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 07:29:04.469650 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:04.469602 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 07:29:04.469893 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:04.469651 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" podUID="2ea0e265-4a3b-46ff-98d1-129054c698df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 07:29:04.503205 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:04.503175 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 07:29:04.521899 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:04.521867 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 07:29:04.522183 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:04.522154 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" podUID="bd13b861-9db1-4e94-bade-fa1a2ca4b04d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 07:29:04.610938 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:04.610045 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 07:29:04.614078 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:04.614012 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 07:29:04.614478 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:04.614452 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 07:29:04.620443 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:04.620412 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 07:29:05.470517 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:05.470478 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 07:29:05.471199 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:05.471140 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" podUID="2ea0e265-4a3b-46ff-98d1-129054c698df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 07:29:05.530596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:05.529694 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 07:29:05.530596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:05.529783 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" podUID="bd13b861-9db1-4e94-bade-fa1a2ca4b04d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 07:29:06.679591 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:06.679348 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:29:06.682590 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:06.681367 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:29:06.683552 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:06.683254 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:29:07.580592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:07.579718 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:29:07.580592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:07.579804 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" podUID="bd13b861-9db1-4e94-bade-fa1a2ca4b04d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:29:08.194019 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:08.193970 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:29:08.195607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:08.194900 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:29:08.198977 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:08.196786 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:29:08.573822 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:08.573789 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a83dbd-cb4df464-2rwhk"] Oct 30 07:29:08.608245 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:08.608200 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a83dbd-66b9d45d75-xr9dv"] Oct 30 07:29:09.187673 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:09.187627 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx"] Oct 30 07:29:09.188370 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:09.188347 4951 cpu_manager.go:414] "containerMap: removing container" podUID="8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b" containerName="main" Oct 30 07:29:09.188370 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:09.188369 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b" containerName="main" Oct 30 07:29:09.188603 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:09.188588 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c4d60ab-c724-48a1-9be3-9f7ba9d9bc8b" containerName="main" Oct 30 07:29:09.193458 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:09.193433 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx" Oct 30 07:29:09.214105 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:09.213653 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-2761aa-serving-cert\"" Oct 30 07:29:09.218010 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:09.217980 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx"] Oct 30 07:29:09.240621 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:09.237310 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f014cb28-67ea-4356-a8a2-846720aaaa6d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx\" (UID: \"f014cb28-67ea-4356-a8a2-846720aaaa6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx" Oct 30 07:29:09.240621 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:09.237432 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f014cb28-67ea-4356-a8a2-846720aaaa6d-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx\" (UID: \"f014cb28-67ea-4356-a8a2-846720aaaa6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx" Oct 30 07:29:09.240621 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:09.237491 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hknrb\" (UniqueName: \"kubernetes.io/projected/f014cb28-67ea-4356-a8a2-846720aaaa6d-kube-api-access-hknrb\") pod \"maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx\" (UID: \"f014cb28-67ea-4356-a8a2-846720aaaa6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx" Oct 30 07:29:09.240621 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:09.237604 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f014cb28-67ea-4356-a8a2-846720aaaa6d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx\" (UID: \"f014cb28-67ea-4356-a8a2-846720aaaa6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx" Oct 30 07:29:09.339519 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:09.339471 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f014cb28-67ea-4356-a8a2-846720aaaa6d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx\" (UID: \"f014cb28-67ea-4356-a8a2-846720aaaa6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx" Oct 30 07:29:09.339746 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:09.339625 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f014cb28-67ea-4356-a8a2-846720aaaa6d-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx\" (UID: \"f014cb28-67ea-4356-a8a2-846720aaaa6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx" Oct 30 07:29:09.339746 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:09.339686 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hknrb\" (UniqueName: \"kubernetes.io/projected/f014cb28-67ea-4356-a8a2-846720aaaa6d-kube-api-access-hknrb\") pod \"maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx\" (UID: \"f014cb28-67ea-4356-a8a2-846720aaaa6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx" Oct 30 07:29:09.339887 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:09.339789 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f014cb28-67ea-4356-a8a2-846720aaaa6d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx\" (UID: \"f014cb28-67ea-4356-a8a2-846720aaaa6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx" Oct 30 07:29:09.340638 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:09.340609 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f014cb28-67ea-4356-a8a2-846720aaaa6d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx\" (UID: \"f014cb28-67ea-4356-a8a2-846720aaaa6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx" Oct 30 07:29:09.345159 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:09.345129 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f014cb28-67ea-4356-a8a2-846720aaaa6d-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx\" (UID: \"f014cb28-67ea-4356-a8a2-846720aaaa6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx" Oct 30 07:29:09.345655 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:09.345623 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f014cb28-67ea-4356-a8a2-846720aaaa6d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx\" (UID: \"f014cb28-67ea-4356-a8a2-846720aaaa6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx" Oct 30 07:29:09.366320 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:09.366280 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hknrb\" (UniqueName: \"kubernetes.io/projected/f014cb28-67ea-4356-a8a2-846720aaaa6d-kube-api-access-hknrb\") pod \"maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx\" (UID: \"f014cb28-67ea-4356-a8a2-846720aaaa6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx" Oct 30 07:29:09.512224 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:09.511929 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx" Oct 30 07:29:09.938455 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:29:09.938401 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf014cb28_67ea_4356_a8a2_846720aaaa6d.slice/crio-118c37b0e2a5b0aa59e7f1d09c41188bf276d4b7c08c1ab2e20d97e993cef54e WatchSource:0}: Error finding container 118c37b0e2a5b0aa59e7f1d09c41188bf276d4b7c08c1ab2e20d97e993cef54e: Status 404 returned error can't find the container with id 118c37b0e2a5b0aa59e7f1d09c41188bf276d4b7c08c1ab2e20d97e993cef54e Oct 30 07:29:09.939356 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:09.939292 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx"] Oct 30 07:29:10.211495 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:10.211415 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx" event={"ID":"f014cb28-67ea-4356-a8a2-846720aaaa6d","Type":"ContainerStarted","Data":"118c37b0e2a5b0aa59e7f1d09c41188bf276d4b7c08c1ab2e20d97e993cef54e"} Oct 30 07:29:11.221722 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:11.221687 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx" event={"ID":"f014cb28-67ea-4356-a8a2-846720aaaa6d","Type":"ContainerStarted","Data":"3b2706b6b07c39a6ada37aebaf8bbc88419f3670b74c73863c1e9bac999ab052"} Oct 30 07:29:11.222249 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:11.221772 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx" Oct 30 07:29:11.269219 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:11.269166 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx" podStartSLOduration=1.9103173660000001 podStartE2EDuration="2.269150972s" totalImagesPullingTime="358.833606ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:29:11.266636455 +0000 UTC m=+1957.609684447" watchObservedRunningTime="2026-10-30 07:29:11.269150972 +0000 UTC m=+1957.612198996" Oct 30 07:29:13.237035 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:13.236998 4951 generic.go:356] "Generic (PLEG): container finished" podID="19bce129-3294-4690-9029-6ddea9c23448" containerID="8587d680a4913102243ec72cd1d22481cb0bf80f83fb2741a6fc8c140ac141c3" exitCode=0 Oct 30 07:29:13.237640 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:13.237051 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" event={"ID":"19bce129-3294-4690-9029-6ddea9c23448","Type":"ContainerDied","Data":"8587d680a4913102243ec72cd1d22481cb0bf80f83fb2741a6fc8c140ac141c3"} Oct 30 07:29:13.237640 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:13.237500 4951 scope.go:122] "RemoveContainer" containerID="8587d680a4913102243ec72cd1d22481cb0bf80f83fb2741a6fc8c140ac141c3" Oct 30 07:29:13.239028 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:13.238991 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:29:13.239139 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:13.239069 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:29:13.239139 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:13.239100 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:29:13.574991 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:13.574929 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-a83dbd-cb4df464-2rwhk" podUID="9d414105-ff0b-4ba9-ad13-a2e2a4c4014e" containerName="payload-processing" containerID="cri-o://d6839ba7dc11bda8faecde4afde2bbdc49a24f1cf107f9e1eb0d766fa25bac3c" gracePeriod=25 Oct 30 07:29:13.609128 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:13.609089 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a83dbd-66b9d45d75-xr9dv" podUID="a96460bf-ba0e-415e-8713-f4da5168996d" containerName="payload-pre-processing" containerID="cri-o://1438fd612c0e84b351a67395f2709f516653daef83278d31c3710c940938d3fc" gracePeriod=25 Oct 30 07:29:13.967621 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:13.965536 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a83dbd-66b9d45d75-xr9dv" Oct 30 07:29:13.968612 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:13.968586 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a96460bf-ba0e-415e-8713-f4da5168996d-kube-api-access-hww2r\" (UniqueName: \"kubernetes.io/projected/a96460bf-ba0e-415e-8713-f4da5168996d-kube-api-access-hww2r\") pod \"a96460bf-ba0e-415e-8713-f4da5168996d\" (UID: \"a96460bf-ba0e-415e-8713-f4da5168996d\") " Oct 30 07:29:13.968760 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:13.968747 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a96460bf-ba0e-415e-8713-f4da5168996d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a96460bf-ba0e-415e-8713-f4da5168996d-plugins-config-volume\") pod \"a96460bf-ba0e-415e-8713-f4da5168996d\" (UID: \"a96460bf-ba0e-415e-8713-f4da5168996d\") " Oct 30 07:29:13.969135 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:13.969111 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a96460bf-ba0e-415e-8713-f4da5168996d-plugins-config-volume" pod "a96460bf-ba0e-415e-8713-f4da5168996d" (UID: "a96460bf-ba0e-415e-8713-f4da5168996d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:29:13.971024 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:13.971001 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a96460bf-ba0e-415e-8713-f4da5168996d-kube-api-access-hww2r" pod "a96460bf-ba0e-415e-8713-f4da5168996d" (UID: "a96460bf-ba0e-415e-8713-f4da5168996d"). InnerVolumeSpecName "kube-api-access-hww2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:29:14.068385 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:14.068352 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a83dbd-cb4df464-2rwhk" Oct 30 07:29:14.070543 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:14.070515 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a96460bf-ba0e-415e-8713-f4da5168996d-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:29:14.070543 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:14.070536 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hww2r\" (UniqueName: \"kubernetes.io/projected/a96460bf-ba0e-415e-8713-f4da5168996d-kube-api-access-hww2r\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:29:14.171705 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:14.171615 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d414105-ff0b-4ba9-ad13-a2e2a4c4014e-kube-api-access-5q924\" (UniqueName: \"kubernetes.io/projected/9d414105-ff0b-4ba9-ad13-a2e2a4c4014e-kube-api-access-5q924\") pod \"9d414105-ff0b-4ba9-ad13-a2e2a4c4014e\" (UID: \"9d414105-ff0b-4ba9-ad13-a2e2a4c4014e\") " Oct 30 07:29:14.171705 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:14.171687 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d414105-ff0b-4ba9-ad13-a2e2a4c4014e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d414105-ff0b-4ba9-ad13-a2e2a4c4014e-plugins-config-volume\") pod \"9d414105-ff0b-4ba9-ad13-a2e2a4c4014e\" (UID: \"9d414105-ff0b-4ba9-ad13-a2e2a4c4014e\") " Oct 30 07:29:14.172184 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:14.172154 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d414105-ff0b-4ba9-ad13-a2e2a4c4014e-plugins-config-volume" pod "9d414105-ff0b-4ba9-ad13-a2e2a4c4014e" (UID: "9d414105-ff0b-4ba9-ad13-a2e2a4c4014e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:29:14.172387 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:14.172365 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d414105-ff0b-4ba9-ad13-a2e2a4c4014e-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:29:14.173890 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:14.173857 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d414105-ff0b-4ba9-ad13-a2e2a4c4014e-kube-api-access-5q924" pod "9d414105-ff0b-4ba9-ad13-a2e2a4c4014e" (UID: "9d414105-ff0b-4ba9-ad13-a2e2a4c4014e"). InnerVolumeSpecName "kube-api-access-5q924". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:29:14.255305 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:14.255269 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" event={"ID":"19bce129-3294-4690-9029-6ddea9c23448","Type":"ContainerStarted","Data":"40dc849ff73cef71f8fbe157121ad49dfbfd3ab7af5c31fcdfceb6751c45d58a"} Oct 30 07:29:14.256768 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:14.256740 4951 generic.go:356] "Generic (PLEG): container finished" podID="9d414105-ff0b-4ba9-ad13-a2e2a4c4014e" containerID="d6839ba7dc11bda8faecde4afde2bbdc49a24f1cf107f9e1eb0d766fa25bac3c" exitCode=0 Oct 30 07:29:14.256956 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:14.256777 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a83dbd-cb4df464-2rwhk" Oct 30 07:29:14.256956 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:14.256817 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a83dbd-cb4df464-2rwhk" event={"ID":"9d414105-ff0b-4ba9-ad13-a2e2a4c4014e","Type":"ContainerDied","Data":"d6839ba7dc11bda8faecde4afde2bbdc49a24f1cf107f9e1eb0d766fa25bac3c"} Oct 30 07:29:14.256956 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:14.256878 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a83dbd-cb4df464-2rwhk" event={"ID":"9d414105-ff0b-4ba9-ad13-a2e2a4c4014e","Type":"ContainerDied","Data":"7cc1116d3bcd08216b79eb87febaffc695b2feaa7bada319f4d69d5d534bfe7c"} Oct 30 07:29:14.257179 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:14.257018 4951 scope.go:122] "RemoveContainer" containerID="d6839ba7dc11bda8faecde4afde2bbdc49a24f1cf107f9e1eb0d766fa25bac3c" Oct 30 07:29:14.259200 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:14.258302 4951 generic.go:356] "Generic (PLEG): container finished" podID="a96460bf-ba0e-415e-8713-f4da5168996d" containerID="1438fd612c0e84b351a67395f2709f516653daef83278d31c3710c940938d3fc" exitCode=0 Oct 30 07:29:14.259200 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:14.258352 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a83dbd-66b9d45d75-xr9dv" event={"ID":"a96460bf-ba0e-415e-8713-f4da5168996d","Type":"ContainerDied","Data":"1438fd612c0e84b351a67395f2709f516653daef83278d31c3710c940938d3fc"} Oct 30 07:29:14.259200 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:14.258381 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a83dbd-66b9d45d75-xr9dv" event={"ID":"a96460bf-ba0e-415e-8713-f4da5168996d","Type":"ContainerDied","Data":"5885a1c66a679db7f024395e82a7add32be2bafd86af0c951b754c44e9c73189"} Oct 30 07:29:14.259200 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:14.258380 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a83dbd-66b9d45d75-xr9dv" Oct 30 07:29:14.269797 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:14.269775 4951 scope.go:122] "RemoveContainer" containerID="d6839ba7dc11bda8faecde4afde2bbdc49a24f1cf107f9e1eb0d766fa25bac3c" Oct 30 07:29:14.270107 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:29:14.270083 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6839ba7dc11bda8faecde4afde2bbdc49a24f1cf107f9e1eb0d766fa25bac3c\": container with ID starting with d6839ba7dc11bda8faecde4afde2bbdc49a24f1cf107f9e1eb0d766fa25bac3c not found: ID does not exist" containerID="d6839ba7dc11bda8faecde4afde2bbdc49a24f1cf107f9e1eb0d766fa25bac3c" Oct 30 07:29:14.270174 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:14.270119 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6839ba7dc11bda8faecde4afde2bbdc49a24f1cf107f9e1eb0d766fa25bac3c"} err="failed to get container status \"d6839ba7dc11bda8faecde4afde2bbdc49a24f1cf107f9e1eb0d766fa25bac3c\": rpc error: code = NotFound desc = could not find container \"d6839ba7dc11bda8faecde4afde2bbdc49a24f1cf107f9e1eb0d766fa25bac3c\": container with ID starting with d6839ba7dc11bda8faecde4afde2bbdc49a24f1cf107f9e1eb0d766fa25bac3c not found: ID does not exist" Oct 30 07:29:14.270174 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:14.270142 4951 scope.go:122] "RemoveContainer" containerID="1438fd612c0e84b351a67395f2709f516653daef83278d31c3710c940938d3fc" Oct 30 07:29:14.273417 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:14.273392 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q924\" (UniqueName: \"kubernetes.io/projected/9d414105-ff0b-4ba9-ad13-a2e2a4c4014e-kube-api-access-5q924\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:29:14.279998 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:14.279978 4951 scope.go:122] "RemoveContainer" containerID="1438fd612c0e84b351a67395f2709f516653daef83278d31c3710c940938d3fc" Oct 30 07:29:14.280315 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:29:14.280291 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1438fd612c0e84b351a67395f2709f516653daef83278d31c3710c940938d3fc\": container with ID starting with 1438fd612c0e84b351a67395f2709f516653daef83278d31c3710c940938d3fc not found: ID does not exist" containerID="1438fd612c0e84b351a67395f2709f516653daef83278d31c3710c940938d3fc" Oct 30 07:29:14.280401 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:14.280326 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1438fd612c0e84b351a67395f2709f516653daef83278d31c3710c940938d3fc"} err="failed to get container status \"1438fd612c0e84b351a67395f2709f516653daef83278d31c3710c940938d3fc\": rpc error: code = NotFound desc = could not find container \"1438fd612c0e84b351a67395f2709f516653daef83278d31c3710c940938d3fc\": container with ID starting with 1438fd612c0e84b351a67395f2709f516653daef83278d31c3710c940938d3fc not found: ID does not exist" Oct 30 07:29:14.350134 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:14.350083 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a83dbd-cb4df464-2rwhk"] Oct 30 07:29:14.353612 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:14.353561 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a83dbd-cb4df464-2rwhk"] Oct 30 07:29:14.399654 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:14.399617 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a83dbd-66b9d45d75-xr9dv"] Oct 30 07:29:14.406097 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:14.406061 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a83dbd-66b9d45d75-xr9dv"] Oct 30 07:29:15.199508 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:15.199475 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:29:15.200643 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:15.200600 4951 patch_prober.go:29] interesting pod/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 07:29:15.200786 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:15.200668 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" podUID="19bce129-3294-4690-9029-6ddea9c23448" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 07:29:15.775646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:15.775605 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d414105-ff0b-4ba9-ad13-a2e2a4c4014e" path="/var/lib/kubelet/pods/9d414105-ff0b-4ba9-ad13-a2e2a4c4014e/volumes" Oct 30 07:29:15.776300 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:15.776279 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a96460bf-ba0e-415e-8713-f4da5168996d" path="/var/lib/kubelet/pods/a96460bf-ba0e-415e-8713-f4da5168996d/volumes" Oct 30 07:29:16.200050 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:16.199961 4951 patch_prober.go:29] interesting pod/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 07:29:16.200050 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:16.200023 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" podUID="19bce129-3294-4690-9029-6ddea9c23448" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 07:29:17.199484 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:17.199449 4951 patch_prober.go:29] interesting pod/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 07:29:17.199945 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:17.199504 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" podUID="19bce129-3294-4690-9029-6ddea9c23448" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 07:29:17.252920 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:17.252881 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx" Oct 30 07:29:18.200236 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.200195 4951 patch_prober.go:29] interesting pod/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 07:29:18.200853 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.200274 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" podUID="19bce129-3294-4690-9029-6ddea9c23448" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 07:29:18.549363 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.549327 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr"] Oct 30 07:29:18.557883 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.557859 4951 cpu_manager.go:414] "containerMap: removing container" podUID="9d414105-ff0b-4ba9-ad13-a2e2a4c4014e" containerName="payload-processing" Oct 30 07:29:18.558080 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.558064 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d414105-ff0b-4ba9-ad13-a2e2a4c4014e" containerName="payload-processing" Oct 30 07:29:18.558268 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.558247 4951 cpu_manager.go:414] "containerMap: removing container" podUID="a96460bf-ba0e-415e-8713-f4da5168996d" containerName="payload-pre-processing" Oct 30 07:29:18.558879 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.558853 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a96460bf-ba0e-415e-8713-f4da5168996d" containerName="payload-pre-processing" Oct 30 07:29:18.559197 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.559180 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d414105-ff0b-4ba9-ad13-a2e2a4c4014e" containerName="payload-processing" Oct 30 07:29:18.559304 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.559293 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="a96460bf-ba0e-415e-8713-f4da5168996d" containerName="payload-pre-processing" Oct 30 07:29:18.563639 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.563621 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:29:18.598046 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.589823 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr"] Oct 30 07:29:18.598046 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.595035 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-4eff1f-openshift-default-dockercfg-mmx2p\"" Oct 30 07:29:18.699814 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.699771 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-istio-data\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:29:18.700153 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.700134 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g4hh\" (UniqueName: \"kubernetes.io/projected/db4df589-43fd-4413-8c51-de66dd1ec2e2-kube-api-access-6g4hh\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:29:18.700309 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.700295 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-istio-envoy\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:29:18.700434 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.700421 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-credential-socket\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:29:18.700613 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.700598 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db4df589-43fd-4413-8c51-de66dd1ec2e2-istiod-ca-cert\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:29:18.700760 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.700747 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-workload-socket\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:29:18.700883 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.700871 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db4df589-43fd-4413-8c51-de66dd1ec2e2-istio-token\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:29:18.701011 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.700996 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db4df589-43fd-4413-8c51-de66dd1ec2e2-istio-podinfo\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:29:18.701203 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.701186 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-workload-certs\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:29:18.803557 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.803452 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-workload-certs\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:29:18.813462 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.804009 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-istio-data\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:29:18.814495 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.814470 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g4hh\" (UniqueName: \"kubernetes.io/projected/db4df589-43fd-4413-8c51-de66dd1ec2e2-kube-api-access-6g4hh\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:29:18.815072 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.815048 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-istio-envoy\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:29:18.819680 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.819655 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-credential-socket\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:29:18.823753 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.823728 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db4df589-43fd-4413-8c51-de66dd1ec2e2-istiod-ca-cert\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:29:18.861178 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.861141 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-workload-socket\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:29:18.861446 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.861424 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db4df589-43fd-4413-8c51-de66dd1ec2e2-istio-token\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:29:18.861661 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.820109 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-credential-socket\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:29:18.861769 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.814088 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-istio-data\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:29:18.861866 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.824551 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db4df589-43fd-4413-8c51-de66dd1ec2e2-istiod-ca-cert\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:29:18.891658 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.844026 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-istio-envoy\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:29:18.891903 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.863679 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db4df589-43fd-4413-8c51-de66dd1ec2e2-istio-podinfo\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:29:18.892268 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.890342 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-workload-socket\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:29:18.892415 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.814336 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-workload-certs\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:29:18.892525 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.814410 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:29:18.910185 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.910153 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db4df589-43fd-4413-8c51-de66dd1ec2e2-istio-podinfo\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:29:18.938099 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.938067 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g4hh\" (UniqueName: \"kubernetes.io/projected/db4df589-43fd-4413-8c51-de66dd1ec2e2-kube-api-access-6g4hh\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:29:18.938846 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:18.938816 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db4df589-43fd-4413-8c51-de66dd1ec2e2-istio-token\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:29:19.195521 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:19.195411 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:29:19.200305 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:19.200276 4951 patch_prober.go:29] interesting pod/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 07:29:19.200752 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:19.200343 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" podUID="19bce129-3294-4690-9029-6ddea9c23448" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 07:29:19.559873 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:19.559838 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr"] Oct 30 07:29:19.563303 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:19.563258 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:29:19.563412 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:19.563359 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:29:19.563486 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:19.563427 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:29:20.205391 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:20.205357 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:29:20.213944 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:20.211894 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:29:20.394414 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:20.394377 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" event={"ID":"db4df589-43fd-4413-8c51-de66dd1ec2e2","Type":"ContainerStarted","Data":"8398656a459109c55a08849f8661b4625027096103bbc31acefc58a5c6273516"} Oct 30 07:29:20.394636 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:20.394441 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" event={"ID":"db4df589-43fd-4413-8c51-de66dd1ec2e2","Type":"ContainerStarted","Data":"6d419dc73c274fc2641dc02d5ffe48d573a391bec27b3400717a761b1c35590d"} Oct 30 07:29:20.445340 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:20.445277 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" podStartSLOduration=2.445259296 podStartE2EDuration="2.445259296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:29:20.442129626 +0000 UTC m=+1966.785177614" watchObservedRunningTime="2026-10-30 07:29:20.445259296 +0000 UTC m=+1966.788307327" Oct 30 07:29:21.196471 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:21.196434 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:29:21.207914 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:21.207881 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:29:21.400649 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:21.400616 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:29:21.401663 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:21.401636 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:29:23.025740 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:29:23.025702 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:29:23.304462 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:23.304374 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb"] Oct 30 07:29:23.304749 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:23.304688 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb" podUID="9f86f9ff-4988-481e-b347-d2314e33fa71" containerName="maas-api" containerID="cri-o://a4d58d6933f26705c93ffc75f76aa9fa7542b85e03f22df5c719c3f2c6f43be9" gracePeriod=30 Oct 30 07:29:23.754604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:23.754309 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb" Oct 30 07:29:23.869613 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:23.863970 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f86f9ff-4988-481e-b347-d2314e33fa71-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f86f9ff-4988-481e-b347-d2314e33fa71-metrics-certs\") pod \"9f86f9ff-4988-481e-b347-d2314e33fa71\" (UID: \"9f86f9ff-4988-481e-b347-d2314e33fa71\") " Oct 30 07:29:23.869613 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:23.864086 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f86f9ff-4988-481e-b347-d2314e33fa71-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f86f9ff-4988-481e-b347-d2314e33fa71-maas-api-tls\") pod \"9f86f9ff-4988-481e-b347-d2314e33fa71\" (UID: \"9f86f9ff-4988-481e-b347-d2314e33fa71\") " Oct 30 07:29:23.869613 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:23.864239 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f86f9ff-4988-481e-b347-d2314e33fa71-kube-api-access-78bn9\" (UniqueName: \"kubernetes.io/projected/9f86f9ff-4988-481e-b347-d2314e33fa71-kube-api-access-78bn9\") pod \"9f86f9ff-4988-481e-b347-d2314e33fa71\" (UID: \"9f86f9ff-4988-481e-b347-d2314e33fa71\") " Oct 30 07:29:23.869613 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:23.864389 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f86f9ff-4988-481e-b347-d2314e33fa71-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f86f9ff-4988-481e-b347-d2314e33fa71-openshift-service-ca\") pod \"9f86f9ff-4988-481e-b347-d2314e33fa71\" (UID: \"9f86f9ff-4988-481e-b347-d2314e33fa71\") " Oct 30 07:29:23.870588 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:23.870216 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f86f9ff-4988-481e-b347-d2314e33fa71-openshift-service-ca" pod "9f86f9ff-4988-481e-b347-d2314e33fa71" (UID: "9f86f9ff-4988-481e-b347-d2314e33fa71"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:29:23.923790 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:23.917699 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f86f9ff-4988-481e-b347-d2314e33fa71-metrics-certs" pod "9f86f9ff-4988-481e-b347-d2314e33fa71" (UID: "9f86f9ff-4988-481e-b347-d2314e33fa71"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:29:23.924091 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:23.924043 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f86f9ff-4988-481e-b347-d2314e33fa71-maas-api-tls" pod "9f86f9ff-4988-481e-b347-d2314e33fa71" (UID: "9f86f9ff-4988-481e-b347-d2314e33fa71"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:29:23.932124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:23.932092 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f86f9ff-4988-481e-b347-d2314e33fa71-kube-api-access-78bn9" pod "9f86f9ff-4988-481e-b347-d2314e33fa71" (UID: "9f86f9ff-4988-481e-b347-d2314e33fa71"). InnerVolumeSpecName "kube-api-access-78bn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:29:23.978526 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:23.978415 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f86f9ff-4988-481e-b347-d2314e33fa71-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:29:23.978526 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:23.978458 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f86f9ff-4988-481e-b347-d2314e33fa71-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:29:23.978526 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:23.978473 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78bn9\" (UniqueName: \"kubernetes.io/projected/9f86f9ff-4988-481e-b347-d2314e33fa71-kube-api-access-78bn9\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:29:23.978526 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:23.978489 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f86f9ff-4988-481e-b347-d2314e33fa71-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:29:24.421739 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:24.421703 4951 generic.go:356] "Generic (PLEG): container finished" podID="9f86f9ff-4988-481e-b347-d2314e33fa71" containerID="a4d58d6933f26705c93ffc75f76aa9fa7542b85e03f22df5c719c3f2c6f43be9" exitCode=0 Oct 30 07:29:24.422204 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:24.421752 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb" Oct 30 07:29:24.422204 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:24.421789 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb" event={"ID":"9f86f9ff-4988-481e-b347-d2314e33fa71","Type":"ContainerDied","Data":"a4d58d6933f26705c93ffc75f76aa9fa7542b85e03f22df5c719c3f2c6f43be9"} Oct 30 07:29:24.422204 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:24.421847 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb" event={"ID":"9f86f9ff-4988-481e-b347-d2314e33fa71","Type":"ContainerDied","Data":"2cd440149cc0ddc1788953a0f6d0936be932dfbbcdcb8a69af187e0fb7959117"} Oct 30 07:29:24.422204 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:24.421876 4951 scope.go:122] "RemoveContainer" containerID="a4d58d6933f26705c93ffc75f76aa9fa7542b85e03f22df5c719c3f2c6f43be9" Oct 30 07:29:24.433537 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:24.433516 4951 scope.go:122] "RemoveContainer" containerID="a4d58d6933f26705c93ffc75f76aa9fa7542b85e03f22df5c719c3f2c6f43be9" Oct 30 07:29:24.433869 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:29:24.433844 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4d58d6933f26705c93ffc75f76aa9fa7542b85e03f22df5c719c3f2c6f43be9\": container with ID starting with a4d58d6933f26705c93ffc75f76aa9fa7542b85e03f22df5c719c3f2c6f43be9 not found: ID does not exist" containerID="a4d58d6933f26705c93ffc75f76aa9fa7542b85e03f22df5c719c3f2c6f43be9" Oct 30 07:29:24.433964 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:24.433879 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4d58d6933f26705c93ffc75f76aa9fa7542b85e03f22df5c719c3f2c6f43be9"} err="failed to get container status \"a4d58d6933f26705c93ffc75f76aa9fa7542b85e03f22df5c719c3f2c6f43be9\": rpc error: code = NotFound desc = could not find container \"a4d58d6933f26705c93ffc75f76aa9fa7542b85e03f22df5c719c3f2c6f43be9\": container with ID starting with a4d58d6933f26705c93ffc75f76aa9fa7542b85e03f22df5c719c3f2c6f43be9 not found: ID does not exist" Oct 30 07:29:24.479278 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:24.479244 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb"] Oct 30 07:29:24.483159 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:24.483135 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a83dbd-5455475d4b-4bcfb"] Oct 30 07:29:24.998502 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:24.998463 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4eff1f-76cb49c85f-vvqsc"] Oct 30 07:29:24.999209 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:24.999190 4951 cpu_manager.go:414] "containerMap: removing container" podUID="9f86f9ff-4988-481e-b347-d2314e33fa71" containerName="maas-api" Oct 30 07:29:24.999209 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:24.999208 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f86f9ff-4988-481e-b347-d2314e33fa71" containerName="maas-api" Oct 30 07:29:24.999439 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:24.999320 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f86f9ff-4988-481e-b347-d2314e33fa71" containerName="maas-api" Oct 30 07:29:25.002432 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:25.002409 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4eff1f-76cb49c85f-vvqsc" Oct 30 07:29:25.022524 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:25.022496 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-4eff1f\"" Oct 30 07:29:25.032606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:25.029303 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4eff1f-76cb49c85f-vvqsc"] Oct 30 07:29:25.206595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:25.158855 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1016f833-117b-4e7a-9678-5dc7b3656bfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4eff1f-76cb49c85f-vvqsc\" (UID: \"1016f833-117b-4e7a-9678-5dc7b3656bfe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4eff1f-76cb49c85f-vvqsc" Oct 30 07:29:25.206595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:25.160039 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-4eff1f-dockercfg-bcdsz\"" Oct 30 07:29:25.270060 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:25.270020 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mqs9\" (UniqueName: \"kubernetes.io/projected/1016f833-117b-4e7a-9678-5dc7b3656bfe-kube-api-access-5mqs9\") pod \"payload-pre-processing-e2e-shared-b-w2-4eff1f-76cb49c85f-vvqsc\" (UID: \"1016f833-117b-4e7a-9678-5dc7b3656bfe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4eff1f-76cb49c85f-vvqsc" Oct 30 07:29:25.312904 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:25.312857 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1016f833-117b-4e7a-9678-5dc7b3656bfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4eff1f-76cb49c85f-vvqsc\" (UID: \"1016f833-117b-4e7a-9678-5dc7b3656bfe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4eff1f-76cb49c85f-vvqsc" Oct 30 07:29:25.334646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:25.314295 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1016f833-117b-4e7a-9678-5dc7b3656bfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4eff1f-76cb49c85f-vvqsc\" (UID: \"1016f833-117b-4e7a-9678-5dc7b3656bfe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4eff1f-76cb49c85f-vvqsc" Oct 30 07:29:25.433596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:25.409251 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4eff1f-76974db5db-jtmws"] Oct 30 07:29:25.433596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:25.430001 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mqs9\" (UniqueName: \"kubernetes.io/projected/1016f833-117b-4e7a-9678-5dc7b3656bfe-kube-api-access-5mqs9\") pod \"payload-pre-processing-e2e-shared-b-w2-4eff1f-76cb49c85f-vvqsc\" (UID: \"1016f833-117b-4e7a-9678-5dc7b3656bfe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4eff1f-76cb49c85f-vvqsc" Oct 30 07:29:25.596177 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:25.596073 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4eff1f-76974db5db-jtmws"] Oct 30 07:29:25.607248 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:25.603703 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4eff1f-76974db5db-jtmws" Oct 30 07:29:25.613146 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:25.612811 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mqs9\" (UniqueName: \"kubernetes.io/projected/1016f833-117b-4e7a-9678-5dc7b3656bfe-kube-api-access-5mqs9\") pod \"payload-pre-processing-e2e-shared-b-w2-4eff1f-76cb49c85f-vvqsc\" (UID: \"1016f833-117b-4e7a-9678-5dc7b3656bfe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4eff1f-76cb49c85f-vvqsc" Oct 30 07:29:25.618142 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:25.618097 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4eff1f-76cb49c85f-vvqsc" Oct 30 07:29:25.683257 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:25.683220 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh9wt\" (UniqueName: \"kubernetes.io/projected/325e2932-31f7-49a9-bac7-d8ac85a4500f-kube-api-access-jh9wt\") pod \"payload-processing-e2e-shared-b-w2-4eff1f-76974db5db-jtmws\" (UID: \"325e2932-31f7-49a9-bac7-d8ac85a4500f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4eff1f-76974db5db-jtmws" Oct 30 07:29:25.683435 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:25.683335 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/325e2932-31f7-49a9-bac7-d8ac85a4500f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-4eff1f-76974db5db-jtmws\" (UID: \"325e2932-31f7-49a9-bac7-d8ac85a4500f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4eff1f-76974db5db-jtmws" Oct 30 07:29:25.775606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:25.775549 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f86f9ff-4988-481e-b347-d2314e33fa71" path="/var/lib/kubelet/pods/9f86f9ff-4988-481e-b347-d2314e33fa71/volumes" Oct 30 07:29:25.784539 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:25.784503 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh9wt\" (UniqueName: \"kubernetes.io/projected/325e2932-31f7-49a9-bac7-d8ac85a4500f-kube-api-access-jh9wt\") pod \"payload-processing-e2e-shared-b-w2-4eff1f-76974db5db-jtmws\" (UID: \"325e2932-31f7-49a9-bac7-d8ac85a4500f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4eff1f-76974db5db-jtmws" Oct 30 07:29:25.784734 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:25.784709 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/325e2932-31f7-49a9-bac7-d8ac85a4500f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-4eff1f-76974db5db-jtmws\" (UID: \"325e2932-31f7-49a9-bac7-d8ac85a4500f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4eff1f-76974db5db-jtmws" Oct 30 07:29:25.785368 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:25.785348 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/325e2932-31f7-49a9-bac7-d8ac85a4500f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-4eff1f-76974db5db-jtmws\" (UID: \"325e2932-31f7-49a9-bac7-d8ac85a4500f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4eff1f-76974db5db-jtmws" Oct 30 07:29:25.810859 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:25.810826 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh9wt\" (UniqueName: \"kubernetes.io/projected/325e2932-31f7-49a9-bac7-d8ac85a4500f-kube-api-access-jh9wt\") pod \"payload-processing-e2e-shared-b-w2-4eff1f-76974db5db-jtmws\" (UID: \"325e2932-31f7-49a9-bac7-d8ac85a4500f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4eff1f-76974db5db-jtmws" Oct 30 07:29:25.990739 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:25.990642 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4eff1f-76cb49c85f-vvqsc"] Oct 30 07:29:26.004596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:25.997624 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4eff1f-76974db5db-jtmws" Oct 30 07:29:26.014653 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:29:26.013791 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1016f833_117b_4e7a_9678_5dc7b3656bfe.slice/crio-d7776adccbefdefd0eab02ced76beab044596a850bf376b60f6243e1ed4d0e05 WatchSource:0}: Error finding container d7776adccbefdefd0eab02ced76beab044596a850bf376b60f6243e1ed4d0e05: Status 404 returned error can't find the container with id d7776adccbefdefd0eab02ced76beab044596a850bf376b60f6243e1ed4d0e05 Oct 30 07:29:26.437348 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:26.436332 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4eff1f-76974db5db-jtmws"] Oct 30 07:29:26.782668 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:26.781298 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4eff1f-76974db5db-jtmws" event={"ID":"325e2932-31f7-49a9-bac7-d8ac85a4500f","Type":"ContainerStarted","Data":"c56d18a69b668639cf907a49f1b2a4d2871dfd734d598e2acebabcc09c9e00e2"} Oct 30 07:29:26.800693 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:26.800636 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4eff1f-76cb49c85f-vvqsc" event={"ID":"1016f833-117b-4e7a-9678-5dc7b3656bfe","Type":"ContainerStarted","Data":"5983b8f0bf5e6d7aec6cf088edd94c1ce059626ead3977cbe7dc7075e21457c3"} Oct 30 07:29:26.800992 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:26.800960 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4eff1f-76cb49c85f-vvqsc" event={"ID":"1016f833-117b-4e7a-9678-5dc7b3656bfe","Type":"ContainerStarted","Data":"d7776adccbefdefd0eab02ced76beab044596a850bf376b60f6243e1ed4d0e05"} Oct 30 07:29:26.801174 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:26.801156 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4eff1f-76cb49c85f-vvqsc" Oct 30 07:29:26.874766 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:26.874694 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4eff1f-76cb49c85f-vvqsc" podStartSLOduration=2.874668695 podStartE2EDuration="2.874668695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:29:26.874670946 +0000 UTC m=+1973.217718911" watchObservedRunningTime="2026-10-30 07:29:26.874668695 +0000 UTC m=+1973.217716661" Oct 30 07:29:27.845674 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:27.845632 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4eff1f-76974db5db-jtmws" event={"ID":"325e2932-31f7-49a9-bac7-d8ac85a4500f","Type":"ContainerStarted","Data":"d3a98c02056dade6e0fe7b9893c4c99d73d5a05b82ed88826f4304a4827590b4"} Oct 30 07:29:27.846426 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:27.846403 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4eff1f-76974db5db-jtmws" Oct 30 07:29:27.890020 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:27.889969 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4eff1f-76974db5db-jtmws" podStartSLOduration=2.889955254 podStartE2EDuration="2.889955254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:29:27.889633938 +0000 UTC m=+1974.232681924" watchObservedRunningTime="2026-10-30 07:29:27.889955254 +0000 UTC m=+1974.233003253" Oct 30 07:29:27.953985 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:27.953688 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx"] Oct 30 07:29:27.953985 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:27.953892 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" podUID="15a9c484-d57b-4968-beab-6d0cdab0e59c" containerName="istio-proxy" containerID="cri-o://8a2299442e9188fde16f146403e54cd2b8a282437436111d937435267adc86b7" gracePeriod=30 Oct 30 07:29:32.275100 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:32.275061 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l"] Oct 30 07:29:32.275938 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:32.275337 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" podUID="43c12ca0-59ed-48ac-8aa7-b72389c8b20d" containerName="istio-proxy" containerID="cri-o://b0f642baf163be09c161669f1b2d24c48f583c66828ad4a059c8ed7689295f19" gracePeriod=30 Oct 30 07:29:33.136479 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.136444 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx"] Oct 30 07:29:33.140815 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.140792 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:29:33.161821 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.161797 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-4b0e81-openshift-default-dockercfg-qq5v7\"" Oct 30 07:29:33.188087 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.188052 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx"] Oct 30 07:29:33.227711 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.227614 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-istio-data\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:29:33.227711 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.227697 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e684243f-7d08-4ae2-b946-7cea93656322-istiod-ca-cert\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:29:33.227963 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.227730 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-workload-socket\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:29:33.227963 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.227864 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbdd8\" (UniqueName: \"kubernetes.io/projected/e684243f-7d08-4ae2-b946-7cea93656322-kube-api-access-bbdd8\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:29:33.227963 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.227922 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-credential-socket\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:29:33.228167 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.227973 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-workload-certs\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:29:33.228167 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.228000 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-istio-envoy\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:29:33.228167 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.228080 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e684243f-7d08-4ae2-b946-7cea93656322-istio-token\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:29:33.228167 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.228161 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e684243f-7d08-4ae2-b946-7cea93656322-istio-podinfo\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:29:33.331601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.331527 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbdd8\" (UniqueName: \"kubernetes.io/projected/e684243f-7d08-4ae2-b946-7cea93656322-kube-api-access-bbdd8\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:29:33.332124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.331662 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-credential-socket\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:29:33.332124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.331728 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-workload-certs\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:29:33.332124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.331759 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-istio-envoy\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:29:33.332124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.331838 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e684243f-7d08-4ae2-b946-7cea93656322-istio-token\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:29:33.332124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.331923 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e684243f-7d08-4ae2-b946-7cea93656322-istio-podinfo\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:29:33.332124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.331981 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-istio-data\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:29:33.332124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.332045 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e684243f-7d08-4ae2-b946-7cea93656322-istiod-ca-cert\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:29:33.332124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.332074 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-workload-socket\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:29:33.365596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.332659 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-workload-socket\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:29:33.365596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.333308 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-credential-socket\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:29:33.365596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.333538 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-workload-certs\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:29:33.365596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.363970 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-istio-data\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:29:33.365596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.364696 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e684243f-7d08-4ae2-b946-7cea93656322-istiod-ca-cert\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:29:33.381595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.378209 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-istio-envoy\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:29:33.390596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.388589 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e684243f-7d08-4ae2-b946-7cea93656322-istio-podinfo\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:29:33.402470 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.395673 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbdd8\" (UniqueName: \"kubernetes.io/projected/e684243f-7d08-4ae2-b946-7cea93656322-kube-api-access-bbdd8\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:29:33.413361 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.413293 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e684243f-7d08-4ae2-b946-7cea93656322-istio-token\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:29:33.462269 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.461868 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:29:33.748993 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.748964 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:29:33.857595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.857496 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-istio-envoy\") pod \"15a9c484-d57b-4968-beab-6d0cdab0e59c\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " Oct 30 07:29:33.857927 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.857901 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-istio-data\") pod \"15a9c484-d57b-4968-beab-6d0cdab0e59c\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " Oct 30 07:29:33.858108 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.858091 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15a9c484-d57b-4968-beab-6d0cdab0e59c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a9c484-d57b-4968-beab-6d0cdab0e59c-istio-podinfo\") pod \"15a9c484-d57b-4968-beab-6d0cdab0e59c\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " Oct 30 07:29:33.858273 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.858256 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15a9c484-d57b-4968-beab-6d0cdab0e59c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a9c484-d57b-4968-beab-6d0cdab0e59c-istiod-ca-cert\") pod \"15a9c484-d57b-4968-beab-6d0cdab0e59c\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " Oct 30 07:29:33.858513 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.858495 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-workload-socket\") pod \"15a9c484-d57b-4968-beab-6d0cdab0e59c\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " Oct 30 07:29:33.858661 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.858645 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-credential-socket\") pod \"15a9c484-d57b-4968-beab-6d0cdab0e59c\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " Oct 30 07:29:33.858821 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.858805 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15a9c484-d57b-4968-beab-6d0cdab0e59c-istio-token\" (UniqueName: \"kubernetes.io/projected/15a9c484-d57b-4968-beab-6d0cdab0e59c-istio-token\") pod \"15a9c484-d57b-4968-beab-6d0cdab0e59c\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " Oct 30 07:29:33.858941 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.858926 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-workload-certs\") pod \"15a9c484-d57b-4968-beab-6d0cdab0e59c\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " Oct 30 07:29:33.859050 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.859032 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15a9c484-d57b-4968-beab-6d0cdab0e59c-kube-api-access-g6nkd\" (UniqueName: \"kubernetes.io/projected/15a9c484-d57b-4968-beab-6d0cdab0e59c-kube-api-access-g6nkd\") pod \"15a9c484-d57b-4968-beab-6d0cdab0e59c\" (UID: \"15a9c484-d57b-4968-beab-6d0cdab0e59c\") " Oct 30 07:29:33.861307 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.861029 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-workload-socket" pod "15a9c484-d57b-4968-beab-6d0cdab0e59c" (UID: "15a9c484-d57b-4968-beab-6d0cdab0e59c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:29:33.861702 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.861471 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-istio-data" pod "15a9c484-d57b-4968-beab-6d0cdab0e59c" (UID: "15a9c484-d57b-4968-beab-6d0cdab0e59c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:29:33.862121 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.861846 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-credential-socket" pod "15a9c484-d57b-4968-beab-6d0cdab0e59c" (UID: "15a9c484-d57b-4968-beab-6d0cdab0e59c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:29:33.863147 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.862259 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-workload-certs" pod "15a9c484-d57b-4968-beab-6d0cdab0e59c" (UID: "15a9c484-d57b-4968-beab-6d0cdab0e59c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:29:33.864536 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.864513 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15a9c484-d57b-4968-beab-6d0cdab0e59c-istiod-ca-cert" pod "15a9c484-d57b-4968-beab-6d0cdab0e59c" (UID: "15a9c484-d57b-4968-beab-6d0cdab0e59c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:29:33.865642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.865606 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a9c484-d57b-4968-beab-6d0cdab0e59c-istio-token" pod "15a9c484-d57b-4968-beab-6d0cdab0e59c" (UID: "15a9c484-d57b-4968-beab-6d0cdab0e59c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:29:33.865882 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.865859 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-istio-envoy" pod "15a9c484-d57b-4968-beab-6d0cdab0e59c" (UID: "15a9c484-d57b-4968-beab-6d0cdab0e59c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:29:33.866021 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.865996 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a9c484-d57b-4968-beab-6d0cdab0e59c-kube-api-access-g6nkd" pod "15a9c484-d57b-4968-beab-6d0cdab0e59c" (UID: "15a9c484-d57b-4968-beab-6d0cdab0e59c"). InnerVolumeSpecName "kube-api-access-g6nkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:29:33.868052 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.868018 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15a9c484-d57b-4968-beab-6d0cdab0e59c-istio-podinfo" pod "15a9c484-d57b-4968-beab-6d0cdab0e59c" (UID: "15a9c484-d57b-4968-beab-6d0cdab0e59c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:29:33.882749 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.882722 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx"] Oct 30 07:29:33.885140 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:29:33.885112 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode684243f_7d08_4ae2_b946_7cea93656322.slice/crio-3be644bfca3483029a89370d935548c07834a02d214fc2e8ca8870e76e2f70df WatchSource:0}: Error finding container 3be644bfca3483029a89370d935548c07834a02d214fc2e8ca8870e76e2f70df: Status 404 returned error can't find the container with id 3be644bfca3483029a89370d935548c07834a02d214fc2e8ca8870e76e2f70df Oct 30 07:29:33.888633 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.888599 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:29:33.888737 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.888676 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:29:33.888737 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.888725 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:29:33.893459 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.893072 4951 generic.go:356] "Generic (PLEG): container finished" podID="15a9c484-d57b-4968-beab-6d0cdab0e59c" containerID="8a2299442e9188fde16f146403e54cd2b8a282437436111d937435267adc86b7" exitCode=0 Oct 30 07:29:33.893459 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.893167 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" event={"ID":"15a9c484-d57b-4968-beab-6d0cdab0e59c","Type":"ContainerDied","Data":"8a2299442e9188fde16f146403e54cd2b8a282437436111d937435267adc86b7"} Oct 30 07:29:33.893459 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.893233 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" event={"ID":"15a9c484-d57b-4968-beab-6d0cdab0e59c","Type":"ContainerDied","Data":"466bd514f83aaedf224b094fd54ad1c00f86c1494eb8e3e39fd1598439aaf69e"} Oct 30 07:29:33.893459 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.893261 4951 scope.go:122] "RemoveContainer" containerID="8a2299442e9188fde16f146403e54cd2b8a282437436111d937435267adc86b7" Oct 30 07:29:33.893749 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.893474 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx" Oct 30 07:29:33.913884 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.913861 4951 scope.go:122] "RemoveContainer" containerID="8a2299442e9188fde16f146403e54cd2b8a282437436111d937435267adc86b7" Oct 30 07:29:33.914199 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:29:33.914180 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a2299442e9188fde16f146403e54cd2b8a282437436111d937435267adc86b7\": container with ID starting with 8a2299442e9188fde16f146403e54cd2b8a282437436111d937435267adc86b7 not found: ID does not exist" containerID="8a2299442e9188fde16f146403e54cd2b8a282437436111d937435267adc86b7" Oct 30 07:29:33.914274 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.914211 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a2299442e9188fde16f146403e54cd2b8a282437436111d937435267adc86b7"} err="failed to get container status \"8a2299442e9188fde16f146403e54cd2b8a282437436111d937435267adc86b7\": rpc error: code = NotFound desc = could not find container \"8a2299442e9188fde16f146403e54cd2b8a282437436111d937435267adc86b7\": container with ID starting with 8a2299442e9188fde16f146403e54cd2b8a282437436111d937435267adc86b7 not found: ID does not exist" Oct 30 07:29:33.961561 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.961528 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:29:33.961561 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.961558 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:29:33.961561 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.961587 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15a9c484-d57b-4968-beab-6d0cdab0e59c-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:29:33.961561 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.961600 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:29:33.961561 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.961613 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6nkd\" (UniqueName: \"kubernetes.io/projected/15a9c484-d57b-4968-beab-6d0cdab0e59c-kube-api-access-g6nkd\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:29:33.961561 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.961622 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:29:33.961561 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.961630 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a9c484-d57b-4968-beab-6d0cdab0e59c-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:29:33.961561 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.961638 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a9c484-d57b-4968-beab-6d0cdab0e59c-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:29:33.961561 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.961646 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a9c484-d57b-4968-beab-6d0cdab0e59c-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:29:33.962172 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.961674 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx"] Oct 30 07:29:33.987887 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:33.987853 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-6w9fx"] Oct 30 07:29:34.904291 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:34.904256 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" event={"ID":"e684243f-7d08-4ae2-b946-7cea93656322","Type":"ContainerStarted","Data":"dbc3d6ada8ab2566f28b0e170bc9eb549a31631e898acb0ea91e6de31d6e7aef"} Oct 30 07:29:34.904729 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:34.904308 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" event={"ID":"e684243f-7d08-4ae2-b946-7cea93656322","Type":"ContainerStarted","Data":"3be644bfca3483029a89370d935548c07834a02d214fc2e8ca8870e76e2f70df"} Oct 30 07:29:34.973968 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:34.973917 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" podStartSLOduration=1.9739015640000002 podStartE2EDuration="1.973901564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:29:34.965335971 +0000 UTC m=+1981.308383938" watchObservedRunningTime="2026-10-30 07:29:34.973901564 +0000 UTC m=+1981.316949528" Oct 30 07:29:35.462560 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:35.462524 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:29:35.468453 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:35.468413 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:29:35.652987 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:35.648146 4951 scope.go:122] "RemoveContainer" containerID="654ebbd2ccf089d6010bd315dc78fd512df44f970b55254ac4a87e16bfb51e6d" Oct 30 07:29:35.674956 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:35.674931 4951 scope.go:122] "RemoveContainer" containerID="4aae1c9712fc15ad3f22d5ba6a59ff13304abeb036564e7f4384974a1b8f1d43" Oct 30 07:29:35.772137 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:35.772107 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15a9c484-d57b-4968-beab-6d0cdab0e59c" path="/var/lib/kubelet/pods/15a9c484-d57b-4968-beab-6d0cdab0e59c/volumes" Oct 30 07:29:35.910644 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:35.910611 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:29:35.911777 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:35.911755 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:29:37.849701 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:37.849530 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4eff1f-76cb49c85f-vvqsc" Oct 30 07:29:37.935676 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:37.935635 4951 generic.go:356] "Generic (PLEG): container finished" podID="43c12ca0-59ed-48ac-8aa7-b72389c8b20d" containerID="b0f642baf163be09c161669f1b2d24c48f583c66828ad4a059c8ed7689295f19" exitCode=0 Oct 30 07:29:37.936668 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:37.936637 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" event={"ID":"43c12ca0-59ed-48ac-8aa7-b72389c8b20d","Type":"ContainerDied","Data":"b0f642baf163be09c161669f1b2d24c48f583c66828ad4a059c8ed7689295f19"} Oct 30 07:29:37.936839 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:37.936696 4951 scope.go:122] "RemoveContainer" containerID="5a01d05c54e68151a209b79f3a527e066560ffd4e8b4da1590cce92e56e84986" Oct 30 07:29:37.983086 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:29:37.983058 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a01d05c54e68151a209b79f3a527e066560ffd4e8b4da1590cce92e56e84986\": container with ID starting with 5a01d05c54e68151a209b79f3a527e066560ffd4e8b4da1590cce92e56e84986 not found: ID does not exist" containerID="5a01d05c54e68151a209b79f3a527e066560ffd4e8b4da1590cce92e56e84986" Oct 30 07:29:37.986544 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:37.986522 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:29:38.025686 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:38.025640 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-workload-certs\") pod \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " Oct 30 07:29:38.025876 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:38.025732 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istio-podinfo\") pod \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " Oct 30 07:29:38.025876 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:38.025774 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-kube-api-access-z8jds\" (UniqueName: \"kubernetes.io/projected/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-kube-api-access-z8jds\") pod \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " Oct 30 07:29:38.026052 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:38.025994 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istio-data\") pod \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " Oct 30 07:29:38.026206 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:38.026006 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-workload-certs" pod "43c12ca0-59ed-48ac-8aa7-b72389c8b20d" (UID: "43c12ca0-59ed-48ac-8aa7-b72389c8b20d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:29:38.026278 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:38.026200 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istio-data" pod "43c12ca0-59ed-48ac-8aa7-b72389c8b20d" (UID: "43c12ca0-59ed-48ac-8aa7-b72389c8b20d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:29:38.026278 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:38.026215 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istiod-ca-cert\") pod \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " Oct 30 07:29:38.026397 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:38.026293 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istio-token\" (UniqueName: \"kubernetes.io/projected/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istio-token\") pod \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " Oct 30 07:29:38.026397 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:38.026327 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istio-envoy\") pod \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " Oct 30 07:29:38.026493 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:38.026402 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-workload-socket\") pod \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " Oct 30 07:29:38.026493 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:38.026457 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-credential-socket\") pod \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\" (UID: \"43c12ca0-59ed-48ac-8aa7-b72389c8b20d\") " Oct 30 07:29:38.026622 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:38.026516 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istiod-ca-cert" pod "43c12ca0-59ed-48ac-8aa7-b72389c8b20d" (UID: "43c12ca0-59ed-48ac-8aa7-b72389c8b20d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:29:38.027158 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:38.026913 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-workload-socket" pod "43c12ca0-59ed-48ac-8aa7-b72389c8b20d" (UID: "43c12ca0-59ed-48ac-8aa7-b72389c8b20d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:29:38.027158 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:38.027087 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-credential-socket" pod "43c12ca0-59ed-48ac-8aa7-b72389c8b20d" (UID: "43c12ca0-59ed-48ac-8aa7-b72389c8b20d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:29:38.027437 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:38.027415 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:29:38.027534 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:38.027442 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:29:38.027534 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:38.027456 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:29:38.027534 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:38.027470 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:29:38.027534 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:38.027484 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:29:38.028721 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:38.028673 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-kube-api-access-z8jds" pod "43c12ca0-59ed-48ac-8aa7-b72389c8b20d" (UID: "43c12ca0-59ed-48ac-8aa7-b72389c8b20d"). InnerVolumeSpecName "kube-api-access-z8jds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:29:38.029061 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:38.029028 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istio-podinfo" pod "43c12ca0-59ed-48ac-8aa7-b72389c8b20d" (UID: "43c12ca0-59ed-48ac-8aa7-b72389c8b20d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:29:38.029510 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:38.029480 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istio-token" pod "43c12ca0-59ed-48ac-8aa7-b72389c8b20d" (UID: "43c12ca0-59ed-48ac-8aa7-b72389c8b20d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:29:38.029747 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:38.029727 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istio-envoy" pod "43c12ca0-59ed-48ac-8aa7-b72389c8b20d" (UID: "43c12ca0-59ed-48ac-8aa7-b72389c8b20d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:29:38.129415 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:38.129323 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:29:38.129415 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:38.129356 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:29:38.129415 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:38.129371 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:29:38.129415 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:38.129383 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8jds\" (UniqueName: \"kubernetes.io/projected/43c12ca0-59ed-48ac-8aa7-b72389c8b20d-kube-api-access-z8jds\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:29:38.855586 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:38.855541 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4eff1f-76974db5db-jtmws" Oct 30 07:29:38.948047 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:38.948011 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" event={"ID":"43c12ca0-59ed-48ac-8aa7-b72389c8b20d","Type":"ContainerDied","Data":"8651ce4905cfadcce9e51f09916b10d0cf1e2185731ba641b4b0fa579aefb9f6"} Oct 30 07:29:38.948252 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:38.948081 4951 scope.go:122] "RemoveContainer" containerID="b0f642baf163be09c161669f1b2d24c48f583c66828ad4a059c8ed7689295f19" Oct 30 07:29:38.948252 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:38.948176 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l" Oct 30 07:29:39.011955 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:39.011924 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l"] Oct 30 07:29:39.028469 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:39.028427 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-a83dbd-openshift-default-6c5bb8888f-j8w2l"] Oct 30 07:29:39.774680 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:39.774644 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43c12ca0-59ed-48ac-8aa7-b72389c8b20d" path="/var/lib/kubelet/pods/43c12ca0-59ed-48ac-8aa7-b72389c8b20d/volumes" Oct 30 07:29:42.829190 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:42.828169 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4"] Oct 30 07:29:42.829742 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:42.829393 4951 cpu_manager.go:414] "containerMap: removing container" podUID="43c12ca0-59ed-48ac-8aa7-b72389c8b20d" containerName="istio-proxy" Oct 30 07:29:42.829742 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:42.829417 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43c12ca0-59ed-48ac-8aa7-b72389c8b20d" containerName="istio-proxy" Oct 30 07:29:42.829742 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:42.829466 4951 cpu_manager.go:414] "containerMap: removing container" podUID="43c12ca0-59ed-48ac-8aa7-b72389c8b20d" containerName="istio-proxy" Oct 30 07:29:42.829742 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:42.829475 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43c12ca0-59ed-48ac-8aa7-b72389c8b20d" containerName="istio-proxy" Oct 30 07:29:42.829742 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:42.829495 4951 cpu_manager.go:414] "containerMap: removing container" podUID="15a9c484-d57b-4968-beab-6d0cdab0e59c" containerName="istio-proxy" Oct 30 07:29:42.829742 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:42.829503 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15a9c484-d57b-4968-beab-6d0cdab0e59c" containerName="istio-proxy" Oct 30 07:29:42.830593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:42.829768 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="43c12ca0-59ed-48ac-8aa7-b72389c8b20d" containerName="istio-proxy" Oct 30 07:29:42.830593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:42.829786 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="43c12ca0-59ed-48ac-8aa7-b72389c8b20d" containerName="istio-proxy" Oct 30 07:29:42.830593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:42.829797 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="15a9c484-d57b-4968-beab-6d0cdab0e59c" containerName="istio-proxy" Oct 30 07:29:42.837727 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:42.836894 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4" Oct 30 07:29:42.856677 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:42.855853 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-4eff1f-serving-cert\"" Oct 30 07:29:42.874681 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:42.874642 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4"] Oct 30 07:29:42.890314 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:42.890267 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f265353-a1b5-4361-bce3-65723e30beec-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4\" (UID: \"3f265353-a1b5-4361-bce3-65723e30beec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4" Oct 30 07:29:42.890841 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:42.890816 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f265353-a1b5-4361-bce3-65723e30beec-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4\" (UID: \"3f265353-a1b5-4361-bce3-65723e30beec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4" Oct 30 07:29:42.891187 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:42.891157 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f265353-a1b5-4361-bce3-65723e30beec-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4\" (UID: \"3f265353-a1b5-4361-bce3-65723e30beec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4" Oct 30 07:29:42.891187 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:42.891204 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8ql4\" (UniqueName: \"kubernetes.io/projected/3f265353-a1b5-4361-bce3-65723e30beec-kube-api-access-l8ql4\") pod \"maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4\" (UID: \"3f265353-a1b5-4361-bce3-65723e30beec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4" Oct 30 07:29:42.992423 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:42.992389 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f265353-a1b5-4361-bce3-65723e30beec-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4\" (UID: \"3f265353-a1b5-4361-bce3-65723e30beec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4" Oct 30 07:29:42.992623 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:42.992428 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8ql4\" (UniqueName: \"kubernetes.io/projected/3f265353-a1b5-4361-bce3-65723e30beec-kube-api-access-l8ql4\") pod \"maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4\" (UID: \"3f265353-a1b5-4361-bce3-65723e30beec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4" Oct 30 07:29:42.992623 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:42.992510 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f265353-a1b5-4361-bce3-65723e30beec-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4\" (UID: \"3f265353-a1b5-4361-bce3-65723e30beec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4" Oct 30 07:29:42.992762 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:42.992628 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f265353-a1b5-4361-bce3-65723e30beec-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4\" (UID: \"3f265353-a1b5-4361-bce3-65723e30beec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4" Oct 30 07:29:42.993175 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:42.993153 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f265353-a1b5-4361-bce3-65723e30beec-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4\" (UID: \"3f265353-a1b5-4361-bce3-65723e30beec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4" Oct 30 07:29:42.995170 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:42.995149 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f265353-a1b5-4361-bce3-65723e30beec-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4\" (UID: \"3f265353-a1b5-4361-bce3-65723e30beec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4" Oct 30 07:29:42.995315 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:42.995291 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f265353-a1b5-4361-bce3-65723e30beec-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4\" (UID: \"3f265353-a1b5-4361-bce3-65723e30beec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4" Oct 30 07:29:43.017092 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:43.017059 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8ql4\" (UniqueName: \"kubernetes.io/projected/3f265353-a1b5-4361-bce3-65723e30beec-kube-api-access-l8ql4\") pod \"maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4\" (UID: \"3f265353-a1b5-4361-bce3-65723e30beec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4" Oct 30 07:29:43.175679 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:43.173263 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4" Oct 30 07:29:43.486955 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:29:43.486926 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f265353_a1b5_4361_bce3_65723e30beec.slice/crio-0a70d69d73adbd86536eaaacbc34b6387270194b25fe61886eca82d847d7b6cf WatchSource:0}: Error finding container 0a70d69d73adbd86536eaaacbc34b6387270194b25fe61886eca82d847d7b6cf: Status 404 returned error can't find the container with id 0a70d69d73adbd86536eaaacbc34b6387270194b25fe61886eca82d847d7b6cf Oct 30 07:29:43.487965 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:43.487937 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4"] Oct 30 07:29:44.118236 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:44.118134 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4" event={"ID":"3f265353-a1b5-4361-bce3-65723e30beec","Type":"ContainerStarted","Data":"2b4b5ad2616946f2c02509de072355338f015abc970179950abff7fc291c7633"} Oct 30 07:29:44.118754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:44.118360 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4" event={"ID":"3f265353-a1b5-4361-bce3-65723e30beec","Type":"ContainerStarted","Data":"0a70d69d73adbd86536eaaacbc34b6387270194b25fe61886eca82d847d7b6cf"} Oct 30 07:29:44.118754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:44.118668 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4" Oct 30 07:29:44.165908 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:44.165835 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4" podStartSLOduration=1.837151151 podStartE2EDuration="2.16581255s" totalImagesPullingTime="328.661399ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:29:44.163681365 +0000 UTC m=+1990.506729351" watchObservedRunningTime="2026-10-30 07:29:44.16581255 +0000 UTC m=+1990.508860518" Oct 30 07:29:50.150180 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:29:50.150142 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4" Oct 30 07:29:53.163951 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:29:53.163912 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:30:00.239586 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:00.239536 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7"] Oct 30 07:30:00.245782 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:00.245718 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7" Oct 30 07:30:00.269983 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:00.269955 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7"] Oct 30 07:30:00.284198 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:00.283324 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:30:00.385253 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:00.385209 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8v2s\" (UniqueName: \"kubernetes.io/projected/2ab5010b-8726-4eb0-b3d7-55b1ba1be01b-kube-api-access-k8v2s\") pod \"maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7\" (UID: \"2ab5010b-8726-4eb0-b3d7-55b1ba1be01b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7" Oct 30 07:30:00.385498 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:00.385419 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ab5010b-8726-4eb0-b3d7-55b1ba1be01b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7\" (UID: \"2ab5010b-8726-4eb0-b3d7-55b1ba1be01b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7" Oct 30 07:30:00.487209 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:00.487163 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8v2s\" (UniqueName: \"kubernetes.io/projected/2ab5010b-8726-4eb0-b3d7-55b1ba1be01b-kube-api-access-k8v2s\") pod \"maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7\" (UID: \"2ab5010b-8726-4eb0-b3d7-55b1ba1be01b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7" Oct 30 07:30:00.487398 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:00.487329 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ab5010b-8726-4eb0-b3d7-55b1ba1be01b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7\" (UID: \"2ab5010b-8726-4eb0-b3d7-55b1ba1be01b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7" Oct 30 07:30:00.488034 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:00.488006 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ab5010b-8726-4eb0-b3d7-55b1ba1be01b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7\" (UID: \"2ab5010b-8726-4eb0-b3d7-55b1ba1be01b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7" Oct 30 07:30:00.514814 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:00.514787 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8v2s\" (UniqueName: \"kubernetes.io/projected/2ab5010b-8726-4eb0-b3d7-55b1ba1be01b-kube-api-access-k8v2s\") pod \"maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7\" (UID: \"2ab5010b-8726-4eb0-b3d7-55b1ba1be01b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7" Oct 30 07:30:00.592350 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:00.592313 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7" Oct 30 07:30:00.915181 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:00.906973 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7"] Oct 30 07:30:01.193451 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:01.193364 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx"] Oct 30 07:30:01.193718 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:01.193629 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx" podUID="c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e" containerName="maas-api" containerID="cri-o://712c8d76e7df07db56315e3a0b8bc9e9dbfe62e87e50e8d6cba13c603bd13b1f" gracePeriod=30 Oct 30 07:30:01.299794 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:01.299762 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7" event={"ID":"2ab5010b-8726-4eb0-b3d7-55b1ba1be01b","Type":"ContainerStarted","Data":"ad0900dca12ba94a641f49df6553656203b1e742e4713ead852ccc6239cf375b"} Oct 30 07:30:01.586602 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:01.579989 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx" Oct 30 07:30:01.722595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:01.722021 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e-metrics-certs\") pod \"c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e\" (UID: \"c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e\") " Oct 30 07:30:01.722595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:01.722094 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e-kube-api-access-8hstl\" (UniqueName: \"kubernetes.io/projected/c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e-kube-api-access-8hstl\") pod \"c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e\" (UID: \"c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e\") " Oct 30 07:30:01.722595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:01.722128 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e-openshift-service-ca\") pod \"c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e\" (UID: \"c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e\") " Oct 30 07:30:01.722595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:01.722459 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e-maas-api-tls\") pod \"c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e\" (UID: \"c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e\") " Oct 30 07:30:01.754596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:01.741010 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e-openshift-service-ca" pod "c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e" (UID: "c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:30:01.827660 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:01.826963 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:30:01.861160 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:01.861118 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e-metrics-certs" pod "c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e" (UID: "c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:30:01.861342 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:01.861302 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e-kube-api-access-8hstl" pod "c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e" (UID: "c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e"). InnerVolumeSpecName "kube-api-access-8hstl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:30:01.861505 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:01.861480 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e-maas-api-tls" pod "c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e" (UID: "c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:30:01.928746 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:01.928676 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:30:01.928746 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:01.928706 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:30:01.928746 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:01.928719 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hstl\" (UniqueName: \"kubernetes.io/projected/c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e-kube-api-access-8hstl\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:30:02.308544 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:02.308511 4951 generic.go:356] "Generic (PLEG): container finished" podID="2ab5010b-8726-4eb0-b3d7-55b1ba1be01b" containerID="46d90929725dc14d0ed8c55392ad4a2a8a1f7c2fdc456991ee4537519fc6fe57" exitCode=60 Oct 30 07:30:02.309033 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:02.308609 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7" event={"ID":"2ab5010b-8726-4eb0-b3d7-55b1ba1be01b","Type":"ContainerDied","Data":"46d90929725dc14d0ed8c55392ad4a2a8a1f7c2fdc456991ee4537519fc6fe57"} Oct 30 07:30:02.309033 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:02.308945 4951 scope.go:122] "RemoveContainer" containerID="46d90929725dc14d0ed8c55392ad4a2a8a1f7c2fdc456991ee4537519fc6fe57" Oct 30 07:30:02.310319 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:02.310276 4951 generic.go:356] "Generic (PLEG): container finished" podID="c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e" containerID="712c8d76e7df07db56315e3a0b8bc9e9dbfe62e87e50e8d6cba13c603bd13b1f" exitCode=0 Oct 30 07:30:02.310445 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:02.310337 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx" Oct 30 07:30:02.310445 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:02.310379 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx" event={"ID":"c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e","Type":"ContainerDied","Data":"712c8d76e7df07db56315e3a0b8bc9e9dbfe62e87e50e8d6cba13c603bd13b1f"} Oct 30 07:30:02.310556 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:02.310451 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx" event={"ID":"c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e","Type":"ContainerDied","Data":"6b9bfea4ba789ebade04db8b880cdc9dc2cee4aabe0ce8b60b8bd7851c17c10e"} Oct 30 07:30:02.310556 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:02.310483 4951 scope.go:122] "RemoveContainer" containerID="712c8d76e7df07db56315e3a0b8bc9e9dbfe62e87e50e8d6cba13c603bd13b1f" Oct 30 07:30:02.327299 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:02.327268 4951 scope.go:122] "RemoveContainer" containerID="712c8d76e7df07db56315e3a0b8bc9e9dbfe62e87e50e8d6cba13c603bd13b1f" Oct 30 07:30:02.327674 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:30:02.327649 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"712c8d76e7df07db56315e3a0b8bc9e9dbfe62e87e50e8d6cba13c603bd13b1f\": container with ID starting with 712c8d76e7df07db56315e3a0b8bc9e9dbfe62e87e50e8d6cba13c603bd13b1f not found: ID does not exist" containerID="712c8d76e7df07db56315e3a0b8bc9e9dbfe62e87e50e8d6cba13c603bd13b1f" Oct 30 07:30:02.327778 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:02.327681 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"712c8d76e7df07db56315e3a0b8bc9e9dbfe62e87e50e8d6cba13c603bd13b1f"} err="failed to get container status \"712c8d76e7df07db56315e3a0b8bc9e9dbfe62e87e50e8d6cba13c603bd13b1f\": rpc error: code = NotFound desc = could not find container \"712c8d76e7df07db56315e3a0b8bc9e9dbfe62e87e50e8d6cba13c603bd13b1f\": container with ID starting with 712c8d76e7df07db56315e3a0b8bc9e9dbfe62e87e50e8d6cba13c603bd13b1f not found: ID does not exist" Oct 30 07:30:02.456026 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:02.455991 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx"] Oct 30 07:30:02.465082 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:02.465031 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5bb165-8464cc67bb-4z8vx"] Oct 30 07:30:03.320482 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:03.320450 4951 generic.go:356] "Generic (PLEG): container finished" podID="2ab5010b-8726-4eb0-b3d7-55b1ba1be01b" containerID="ff92c62b3cdeb7ed1210414bd31030e6e315083a2473fe52bfe3791f0631b7e1" exitCode=60 Oct 30 07:30:03.320970 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:03.320534 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7" event={"ID":"2ab5010b-8726-4eb0-b3d7-55b1ba1be01b","Type":"ContainerDied","Data":"ff92c62b3cdeb7ed1210414bd31030e6e315083a2473fe52bfe3791f0631b7e1"} Oct 30 07:30:03.320970 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:03.320612 4951 scope.go:122] "RemoveContainer" containerID="46d90929725dc14d0ed8c55392ad4a2a8a1f7c2fdc456991ee4537519fc6fe57" Oct 30 07:30:03.320970 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:03.320937 4951 scope.go:122] "RemoveContainer" containerID="ff92c62b3cdeb7ed1210414bd31030e6e315083a2473fe52bfe3791f0631b7e1" Oct 30 07:30:03.321292 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:30:03.321270 4951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7_odh-ai-gateway-infra(2ab5010b-8726-4eb0-b3d7-55b1ba1be01b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7" podUID="2ab5010b-8726-4eb0-b3d7-55b1ba1be01b" Oct 30 07:30:03.772121 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:03.772088 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e" path="/var/lib/kubelet/pods/c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e/volumes" Oct 30 07:30:04.331924 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:04.331893 4951 scope.go:122] "RemoveContainer" containerID="ff92c62b3cdeb7ed1210414bd31030e6e315083a2473fe52bfe3791f0631b7e1" Oct 30 07:30:04.332340 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:30:04.332302 4951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7_odh-ai-gateway-infra(2ab5010b-8726-4eb0-b3d7-55b1ba1be01b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7" podUID="2ab5010b-8726-4eb0-b3d7-55b1ba1be01b" Oct 30 07:30:05.544498 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:05.544455 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8"] Oct 30 07:30:05.545195 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:05.545162 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" podUID="19bce129-3294-4690-9029-6ddea9c23448" containerName="istio-proxy" containerID="cri-o://40dc849ff73cef71f8fbe157121ad49dfbfd3ab7af5c31fcdfceb6751c45d58a" gracePeriod=30 Oct 30 07:30:09.500833 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:09.500756 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm"] Oct 30 07:30:09.501236 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:09.501012 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" podUID="4506abe9-6f43-4070-9b7b-64631f5be137" containerName="istio-proxy" containerID="cri-o://de7e9f9678a375bbdad9114fd00232caddcbc9c021d025f225dfc980640c68d3" gracePeriod=30 Oct 30 07:30:10.850195 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:10.850160 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:30:10.912422 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:10.912384 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-credential-socket\") pod \"19bce129-3294-4690-9029-6ddea9c23448\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " Oct 30 07:30:10.912632 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:10.912445 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-workload-socket\") pod \"19bce129-3294-4690-9029-6ddea9c23448\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " Oct 30 07:30:10.912632 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:10.912507 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19bce129-3294-4690-9029-6ddea9c23448-kube-api-access-tvmw4\" (UniqueName: \"kubernetes.io/projected/19bce129-3294-4690-9029-6ddea9c23448-kube-api-access-tvmw4\") pod \"19bce129-3294-4690-9029-6ddea9c23448\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " Oct 30 07:30:10.912780 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:10.912661 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19bce129-3294-4690-9029-6ddea9c23448-istio-token\" (UniqueName: \"kubernetes.io/projected/19bce129-3294-4690-9029-6ddea9c23448-istio-token\") pod \"19bce129-3294-4690-9029-6ddea9c23448\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " Oct 30 07:30:10.912780 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:10.912695 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-credential-socket" pod "19bce129-3294-4690-9029-6ddea9c23448" (UID: "19bce129-3294-4690-9029-6ddea9c23448"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:30:10.912780 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:10.912743 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-workload-certs\") pod \"19bce129-3294-4690-9029-6ddea9c23448\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " Oct 30 07:30:10.912780 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:10.912775 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-istio-data\") pod \"19bce129-3294-4690-9029-6ddea9c23448\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " Oct 30 07:30:10.912988 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:10.912805 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19bce129-3294-4690-9029-6ddea9c23448-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19bce129-3294-4690-9029-6ddea9c23448-istiod-ca-cert\") pod \"19bce129-3294-4690-9029-6ddea9c23448\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " Oct 30 07:30:10.912988 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:10.912886 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19bce129-3294-4690-9029-6ddea9c23448-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19bce129-3294-4690-9029-6ddea9c23448-istio-podinfo\") pod \"19bce129-3294-4690-9029-6ddea9c23448\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " Oct 30 07:30:10.912988 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:10.912931 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-istio-envoy\") pod \"19bce129-3294-4690-9029-6ddea9c23448\" (UID: \"19bce129-3294-4690-9029-6ddea9c23448\") " Oct 30 07:30:10.912988 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:10.912929 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-workload-socket" pod "19bce129-3294-4690-9029-6ddea9c23448" (UID: "19bce129-3294-4690-9029-6ddea9c23448"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:30:10.913613 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:10.913503 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19bce129-3294-4690-9029-6ddea9c23448-istiod-ca-cert" pod "19bce129-3294-4690-9029-6ddea9c23448" (UID: "19bce129-3294-4690-9029-6ddea9c23448"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:30:10.913613 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:10.913527 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:30:10.913613 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:10.913550 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:30:10.913880 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:10.913791 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-workload-certs" pod "19bce129-3294-4690-9029-6ddea9c23448" (UID: "19bce129-3294-4690-9029-6ddea9c23448"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:30:10.914090 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:10.914065 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-istio-data" pod "19bce129-3294-4690-9029-6ddea9c23448" (UID: "19bce129-3294-4690-9029-6ddea9c23448"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:30:10.915300 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:10.915275 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19bce129-3294-4690-9029-6ddea9c23448-kube-api-access-tvmw4" pod "19bce129-3294-4690-9029-6ddea9c23448" (UID: "19bce129-3294-4690-9029-6ddea9c23448"). InnerVolumeSpecName "kube-api-access-tvmw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:30:10.915972 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:10.915950 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-istio-envoy" pod "19bce129-3294-4690-9029-6ddea9c23448" (UID: "19bce129-3294-4690-9029-6ddea9c23448"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:30:10.916093 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:10.916048 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19bce129-3294-4690-9029-6ddea9c23448-istio-podinfo" pod "19bce129-3294-4690-9029-6ddea9c23448" (UID: "19bce129-3294-4690-9029-6ddea9c23448"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:30:10.917441 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:10.917421 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19bce129-3294-4690-9029-6ddea9c23448-istio-token" pod "19bce129-3294-4690-9029-6ddea9c23448" (UID: "19bce129-3294-4690-9029-6ddea9c23448"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:30:11.014739 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.014687 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvmw4\" (UniqueName: \"kubernetes.io/projected/19bce129-3294-4690-9029-6ddea9c23448-kube-api-access-tvmw4\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:30:11.014739 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.014729 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19bce129-3294-4690-9029-6ddea9c23448-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:30:11.014739 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.014743 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:30:11.015008 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.014754 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:30:11.015008 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.014766 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19bce129-3294-4690-9029-6ddea9c23448-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:30:11.015008 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.014777 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19bce129-3294-4690-9029-6ddea9c23448-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:30:11.015008 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.014787 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19bce129-3294-4690-9029-6ddea9c23448-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:30:11.337457 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.336653 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2"] Oct 30 07:30:11.337721 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.337697 4951 cpu_manager.go:414] "containerMap: removing container" podUID="c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e" containerName="maas-api" Oct 30 07:30:11.337838 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.337723 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e" containerName="maas-api" Oct 30 07:30:11.337838 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.337752 4951 cpu_manager.go:414] "containerMap: removing container" podUID="19bce129-3294-4690-9029-6ddea9c23448" containerName="istio-proxy" Oct 30 07:30:11.337838 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.337760 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19bce129-3294-4690-9029-6ddea9c23448" containerName="istio-proxy" Oct 30 07:30:11.337838 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.337791 4951 cpu_manager.go:414] "containerMap: removing container" podUID="19bce129-3294-4690-9029-6ddea9c23448" containerName="istio-proxy" Oct 30 07:30:11.337838 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.337799 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19bce129-3294-4690-9029-6ddea9c23448" containerName="istio-proxy" Oct 30 07:30:11.338107 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.337973 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7be83ae-eff0-4c63-8e2c-9c3748ea7c2e" containerName="maas-api" Oct 30 07:30:11.338107 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.337992 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="19bce129-3294-4690-9029-6ddea9c23448" containerName="istio-proxy" Oct 30 07:30:11.338107 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.338010 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="19bce129-3294-4690-9029-6ddea9c23448" containerName="istio-proxy" Oct 30 07:30:11.344886 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.344856 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:11.366365 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.365747 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2"] Oct 30 07:30:11.373796 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.373769 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-46dac2-openshift-default-dockercfg-5jfqt\"" Oct 30 07:30:11.420476 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.420443 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istio-podinfo\") pod \"e2e-autores-46dac2-openshift-default-5694949599-pr4k2\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:11.420676 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.420627 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istio-token\") pod \"e2e-autores-46dac2-openshift-default-5694949599-pr4k2\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:11.420763 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.420692 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-credential-socket\") pod \"e2e-autores-46dac2-openshift-default-5694949599-pr4k2\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:11.420763 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.420741 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r2xl\" (UniqueName: \"kubernetes.io/projected/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-kube-api-access-8r2xl\") pod \"e2e-autores-46dac2-openshift-default-5694949599-pr4k2\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:11.420847 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.420824 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istiod-ca-cert\") pod \"e2e-autores-46dac2-openshift-default-5694949599-pr4k2\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:11.420883 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.420857 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-workload-socket\") pod \"e2e-autores-46dac2-openshift-default-5694949599-pr4k2\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:11.420883 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.420874 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istio-data\") pod \"e2e-autores-46dac2-openshift-default-5694949599-pr4k2\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:11.420992 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.420977 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-workload-certs\") pod \"e2e-autores-46dac2-openshift-default-5694949599-pr4k2\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:11.421076 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.421063 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istio-envoy\") pod \"e2e-autores-46dac2-openshift-default-5694949599-pr4k2\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:11.468022 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.467981 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t"] Oct 30 07:30:11.472541 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.472517 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t" Oct 30 07:30:11.477652 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.477617 4951 generic.go:356] "Generic (PLEG): container finished" podID="19bce129-3294-4690-9029-6ddea9c23448" containerID="40dc849ff73cef71f8fbe157121ad49dfbfd3ab7af5c31fcdfceb6751c45d58a" exitCode=0 Oct 30 07:30:11.477833 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.477685 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" Oct 30 07:30:11.477833 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.477684 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" event={"ID":"19bce129-3294-4690-9029-6ddea9c23448","Type":"ContainerDied","Data":"40dc849ff73cef71f8fbe157121ad49dfbfd3ab7af5c31fcdfceb6751c45d58a"} Oct 30 07:30:11.477962 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.477835 4951 scope.go:122] "RemoveContainer" containerID="40dc849ff73cef71f8fbe157121ad49dfbfd3ab7af5c31fcdfceb6751c45d58a" Oct 30 07:30:11.478735 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.478705 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8" event={"ID":"19bce129-3294-4690-9029-6ddea9c23448","Type":"ContainerDied","Data":"d10f01d89becc026631831c1510008fcf0f1fa9223655410d5d36505e3a4d167"} Oct 30 07:30:11.497035 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.496985 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-bd38cf-serving-cert\"" Oct 30 07:30:11.544609 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.530448 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/910285f4-8b96-44a7-a0e5-542f04538151-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t\" (UID: \"910285f4-8b96-44a7-a0e5-542f04538151\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t" Oct 30 07:30:11.544609 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.530519 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istio-envoy\") pod \"e2e-autores-46dac2-openshift-default-5694949599-pr4k2\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:11.544609 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.530586 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/910285f4-8b96-44a7-a0e5-542f04538151-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t\" (UID: \"910285f4-8b96-44a7-a0e5-542f04538151\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t" Oct 30 07:30:11.544609 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.530636 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/910285f4-8b96-44a7-a0e5-542f04538151-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t\" (UID: \"910285f4-8b96-44a7-a0e5-542f04538151\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t" Oct 30 07:30:11.544609 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.530701 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istio-podinfo\") pod \"e2e-autores-46dac2-openshift-default-5694949599-pr4k2\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:11.544609 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.530814 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk7bk\" (UniqueName: \"kubernetes.io/projected/910285f4-8b96-44a7-a0e5-542f04538151-kube-api-access-jk7bk\") pod \"maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t\" (UID: \"910285f4-8b96-44a7-a0e5-542f04538151\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t" Oct 30 07:30:11.544609 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.530842 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istio-token\") pod \"e2e-autores-46dac2-openshift-default-5694949599-pr4k2\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:11.544609 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.530912 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-credential-socket\") pod \"e2e-autores-46dac2-openshift-default-5694949599-pr4k2\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:11.544609 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.530961 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r2xl\" (UniqueName: \"kubernetes.io/projected/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-kube-api-access-8r2xl\") pod \"e2e-autores-46dac2-openshift-default-5694949599-pr4k2\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:11.544609 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.531015 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istiod-ca-cert\") pod \"e2e-autores-46dac2-openshift-default-5694949599-pr4k2\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:11.544609 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.531046 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-workload-socket\") pod \"e2e-autores-46dac2-openshift-default-5694949599-pr4k2\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:11.544609 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.531072 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istio-data\") pod \"e2e-autores-46dac2-openshift-default-5694949599-pr4k2\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:11.544609 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.531115 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-workload-certs\") pod \"e2e-autores-46dac2-openshift-default-5694949599-pr4k2\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:11.544609 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.531647 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-workload-certs\") pod \"e2e-autores-46dac2-openshift-default-5694949599-pr4k2\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:11.755131 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.753047 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t"] Oct 30 07:30:11.816907 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.816094 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/910285f4-8b96-44a7-a0e5-542f04538151-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t\" (UID: \"910285f4-8b96-44a7-a0e5-542f04538151\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t" Oct 30 07:30:11.816907 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.816180 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/910285f4-8b96-44a7-a0e5-542f04538151-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t\" (UID: \"910285f4-8b96-44a7-a0e5-542f04538151\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t" Oct 30 07:30:11.816907 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.816238 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/910285f4-8b96-44a7-a0e5-542f04538151-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t\" (UID: \"910285f4-8b96-44a7-a0e5-542f04538151\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t" Oct 30 07:30:11.816907 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.816396 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk7bk\" (UniqueName: \"kubernetes.io/projected/910285f4-8b96-44a7-a0e5-542f04538151-kube-api-access-jk7bk\") pod \"maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t\" (UID: \"910285f4-8b96-44a7-a0e5-542f04538151\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t" Oct 30 07:30:11.845694 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.830552 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/910285f4-8b96-44a7-a0e5-542f04538151-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t\" (UID: \"910285f4-8b96-44a7-a0e5-542f04538151\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t" Oct 30 07:30:11.845694 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.838160 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/910285f4-8b96-44a7-a0e5-542f04538151-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t\" (UID: \"910285f4-8b96-44a7-a0e5-542f04538151\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t" Oct 30 07:30:11.850308 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.850277 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istiod-ca-cert\") pod \"e2e-autores-46dac2-openshift-default-5694949599-pr4k2\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:11.853560 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.853415 4951 scope.go:122] "RemoveContainer" containerID="8587d680a4913102243ec72cd1d22481cb0bf80f83fb2741a6fc8c140ac141c3" Oct 30 07:30:11.856796 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.856763 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8"] Oct 30 07:30:11.863968 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.863931 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk7bk\" (UniqueName: \"kubernetes.io/projected/910285f4-8b96-44a7-a0e5-542f04538151-kube-api-access-jk7bk\") pod \"maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t\" (UID: \"910285f4-8b96-44a7-a0e5-542f04538151\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t" Oct 30 07:30:11.864205 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.864177 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istio-envoy\") pod \"e2e-autores-46dac2-openshift-default-5694949599-pr4k2\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:11.864205 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.864195 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-workload-socket\") pod \"e2e-autores-46dac2-openshift-default-5694949599-pr4k2\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:11.864372 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.864298 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-credential-socket\") pod \"e2e-autores-46dac2-openshift-default-5694949599-pr4k2\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:11.864372 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.864330 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istio-data\") pod \"e2e-autores-46dac2-openshift-default-5694949599-pr4k2\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:11.864757 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.864733 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istio-podinfo\") pod \"e2e-autores-46dac2-openshift-default-5694949599-pr4k2\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:11.866267 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.866223 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/910285f4-8b96-44a7-a0e5-542f04538151-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t\" (UID: \"910285f4-8b96-44a7-a0e5-542f04538151\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t" Oct 30 07:30:11.867700 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.867678 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r2xl\" (UniqueName: \"kubernetes.io/projected/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-kube-api-access-8r2xl\") pod \"e2e-autores-46dac2-openshift-default-5694949599-pr4k2\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:11.868651 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.868631 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-2ddf97-openshift-default-79c5b766cc-frfg8"] Oct 30 07:30:11.874765 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.874730 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istio-token\") pod \"e2e-autores-46dac2-openshift-default-5694949599-pr4k2\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:11.899414 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.899389 4951 scope.go:122] "RemoveContainer" containerID="40dc849ff73cef71f8fbe157121ad49dfbfd3ab7af5c31fcdfceb6751c45d58a" Oct 30 07:30:11.899841 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:30:11.899815 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40dc849ff73cef71f8fbe157121ad49dfbfd3ab7af5c31fcdfceb6751c45d58a\": container with ID starting with 40dc849ff73cef71f8fbe157121ad49dfbfd3ab7af5c31fcdfceb6751c45d58a not found: ID does not exist" containerID="40dc849ff73cef71f8fbe157121ad49dfbfd3ab7af5c31fcdfceb6751c45d58a" Oct 30 07:30:11.899910 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.899852 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40dc849ff73cef71f8fbe157121ad49dfbfd3ab7af5c31fcdfceb6751c45d58a"} err="failed to get container status \"40dc849ff73cef71f8fbe157121ad49dfbfd3ab7af5c31fcdfceb6751c45d58a\": rpc error: code = NotFound desc = could not find container \"40dc849ff73cef71f8fbe157121ad49dfbfd3ab7af5c31fcdfceb6751c45d58a\": container with ID starting with 40dc849ff73cef71f8fbe157121ad49dfbfd3ab7af5c31fcdfceb6751c45d58a not found: ID does not exist" Oct 30 07:30:11.899910 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.899872 4951 scope.go:122] "RemoveContainer" containerID="8587d680a4913102243ec72cd1d22481cb0bf80f83fb2741a6fc8c140ac141c3" Oct 30 07:30:11.900186 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:30:11.900169 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8587d680a4913102243ec72cd1d22481cb0bf80f83fb2741a6fc8c140ac141c3\": container with ID starting with 8587d680a4913102243ec72cd1d22481cb0bf80f83fb2741a6fc8c140ac141c3 not found: ID does not exist" containerID="8587d680a4913102243ec72cd1d22481cb0bf80f83fb2741a6fc8c140ac141c3" Oct 30 07:30:11.900239 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.900196 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8587d680a4913102243ec72cd1d22481cb0bf80f83fb2741a6fc8c140ac141c3"} err="failed to get container status \"8587d680a4913102243ec72cd1d22481cb0bf80f83fb2741a6fc8c140ac141c3\": rpc error: code = NotFound desc = could not find container \"8587d680a4913102243ec72cd1d22481cb0bf80f83fb2741a6fc8c140ac141c3\": container with ID starting with 8587d680a4913102243ec72cd1d22481cb0bf80f83fb2741a6fc8c140ac141c3 not found: ID does not exist" Oct 30 07:30:11.963134 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:11.963044 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:12.130533 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:12.130214 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t" Oct 30 07:30:12.287894 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:30:12.287860 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ca6eb8_437b_4a4f_abea_d2ccf7243c8d.slice/crio-653dd7c9dfecfb2de2aa30865ec3b508caae9b696be8f3cfaab281c9d43cd1ec WatchSource:0}: Error finding container 653dd7c9dfecfb2de2aa30865ec3b508caae9b696be8f3cfaab281c9d43cd1ec: Status 404 returned error can't find the container with id 653dd7c9dfecfb2de2aa30865ec3b508caae9b696be8f3cfaab281c9d43cd1ec Oct 30 07:30:12.289735 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:12.289709 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2"] Oct 30 07:30:12.291125 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:12.291090 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:30:12.291212 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:12.291156 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:30:12.291212 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:12.291185 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:30:12.501243 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:12.482236 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t"] Oct 30 07:30:12.563851 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:12.563522 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t" event={"ID":"910285f4-8b96-44a7-a0e5-542f04538151","Type":"ContainerStarted","Data":"399b1319b13c1c7f00adbd141736f92ad070f77d178dbc2f1a5c218d97aeec6f"} Oct 30 07:30:12.568745 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:12.568693 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" event={"ID":"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d","Type":"ContainerStarted","Data":"9bf5fc5978cdaa51c9de177715897074265891c9c8d5a374592115ff4b9e500b"} Oct 30 07:30:12.568940 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:12.568753 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" event={"ID":"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d","Type":"ContainerStarted","Data":"653dd7c9dfecfb2de2aa30865ec3b508caae9b696be8f3cfaab281c9d43cd1ec"} Oct 30 07:30:12.625617 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:12.622788 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" podStartSLOduration=1.622766753 podStartE2EDuration="1.622766753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:12.619526957 +0000 UTC m=+2018.962574927" watchObservedRunningTime="2026-10-30 07:30:12.622766753 +0000 UTC m=+2018.965814719" Oct 30 07:30:12.963655 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:12.963619 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:13.610416 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:13.609758 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t" event={"ID":"910285f4-8b96-44a7-a0e5-542f04538151","Type":"ContainerStarted","Data":"bce973ee83c8fcb6e8b525b7478edbcf1a5b595e4c63b85cd0ea6c6a3f98b155"} Oct 30 07:30:13.610416 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:13.610004 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t" Oct 30 07:30:13.659172 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:13.659111 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t" podStartSLOduration=2.2710093479999998 podStartE2EDuration="2.65909257s" totalImagesPullingTime="388.083222ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:13.655278487 +0000 UTC m=+2019.998326453" watchObservedRunningTime="2026-10-30 07:30:13.65909257 +0000 UTC m=+2020.002140536" Oct 30 07:30:13.821443 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:13.821140 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19bce129-3294-4690-9029-6ddea9c23448" path="/var/lib/kubelet/pods/19bce129-3294-4690-9029-6ddea9c23448/volumes" Oct 30 07:30:13.971235 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:13.971159 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:14.618204 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:14.618170 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:14.619261 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:14.619234 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:14.841055 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:14.841021 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:30:14.917356 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:14.917317 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4506abe9-6f43-4070-9b7b-64631f5be137-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4506abe9-6f43-4070-9b7b-64631f5be137-istio-podinfo\") pod \"4506abe9-6f43-4070-9b7b-64631f5be137\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " Oct 30 07:30:14.917567 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:14.917408 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-istio-data\") pod \"4506abe9-6f43-4070-9b7b-64631f5be137\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " Oct 30 07:30:14.917567 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:14.917450 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4506abe9-6f43-4070-9b7b-64631f5be137-kube-api-access-x5dck\" (UniqueName: \"kubernetes.io/projected/4506abe9-6f43-4070-9b7b-64631f5be137-kube-api-access-x5dck\") pod \"4506abe9-6f43-4070-9b7b-64631f5be137\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " Oct 30 07:30:14.917725 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:14.917699 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4506abe9-6f43-4070-9b7b-64631f5be137-istio-token\" (UniqueName: \"kubernetes.io/projected/4506abe9-6f43-4070-9b7b-64631f5be137-istio-token\") pod \"4506abe9-6f43-4070-9b7b-64631f5be137\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " Oct 30 07:30:14.917789 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:14.917738 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-workload-certs\") pod \"4506abe9-6f43-4070-9b7b-64631f5be137\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " Oct 30 07:30:14.917847 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:14.917807 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-istio-envoy\") pod \"4506abe9-6f43-4070-9b7b-64631f5be137\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " Oct 30 07:30:14.917897 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:14.917842 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-workload-socket\") pod \"4506abe9-6f43-4070-9b7b-64631f5be137\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " Oct 30 07:30:14.917952 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:14.917918 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-credential-socket\") pod \"4506abe9-6f43-4070-9b7b-64631f5be137\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " Oct 30 07:30:14.918004 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:14.917966 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4506abe9-6f43-4070-9b7b-64631f5be137-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4506abe9-6f43-4070-9b7b-64631f5be137-istiod-ca-cert\") pod \"4506abe9-6f43-4070-9b7b-64631f5be137\" (UID: \"4506abe9-6f43-4070-9b7b-64631f5be137\") " Oct 30 07:30:14.918997 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:14.918961 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4506abe9-6f43-4070-9b7b-64631f5be137-istiod-ca-cert" pod "4506abe9-6f43-4070-9b7b-64631f5be137" (UID: "4506abe9-6f43-4070-9b7b-64631f5be137"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:30:14.919262 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:14.919237 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-istio-data" pod "4506abe9-6f43-4070-9b7b-64631f5be137" (UID: "4506abe9-6f43-4070-9b7b-64631f5be137"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:30:14.920614 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:14.920562 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-workload-certs" pod "4506abe9-6f43-4070-9b7b-64631f5be137" (UID: "4506abe9-6f43-4070-9b7b-64631f5be137"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:30:14.920802 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:14.920783 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-credential-socket" pod "4506abe9-6f43-4070-9b7b-64631f5be137" (UID: "4506abe9-6f43-4070-9b7b-64631f5be137"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:30:14.920975 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:14.920948 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-workload-socket" pod "4506abe9-6f43-4070-9b7b-64631f5be137" (UID: "4506abe9-6f43-4070-9b7b-64631f5be137"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:30:14.922744 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:14.922511 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4506abe9-6f43-4070-9b7b-64631f5be137-kube-api-access-x5dck" pod "4506abe9-6f43-4070-9b7b-64631f5be137" (UID: "4506abe9-6f43-4070-9b7b-64631f5be137"). InnerVolumeSpecName "kube-api-access-x5dck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:30:14.922744 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:14.922680 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4506abe9-6f43-4070-9b7b-64631f5be137-istio-podinfo" pod "4506abe9-6f43-4070-9b7b-64631f5be137" (UID: "4506abe9-6f43-4070-9b7b-64631f5be137"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:30:14.923925 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:14.923896 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-istio-envoy" pod "4506abe9-6f43-4070-9b7b-64631f5be137" (UID: "4506abe9-6f43-4070-9b7b-64631f5be137"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:30:14.924335 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:14.924304 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4506abe9-6f43-4070-9b7b-64631f5be137-istio-token" pod "4506abe9-6f43-4070-9b7b-64631f5be137" (UID: "4506abe9-6f43-4070-9b7b-64631f5be137"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:30:15.019830 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:15.019797 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4506abe9-6f43-4070-9b7b-64631f5be137-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:30:15.019830 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:15.019826 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:30:15.019830 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:15.019836 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:30:15.020284 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:15.019844 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:30:15.020284 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:15.019853 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:30:15.020284 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:15.019863 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4506abe9-6f43-4070-9b7b-64631f5be137-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:30:15.020284 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:15.019872 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4506abe9-6f43-4070-9b7b-64631f5be137-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:30:15.020284 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:15.019880 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4506abe9-6f43-4070-9b7b-64631f5be137-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:30:15.020284 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:15.019889 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5dck\" (UniqueName: \"kubernetes.io/projected/4506abe9-6f43-4070-9b7b-64631f5be137-kube-api-access-x5dck\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:30:15.627268 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:15.627170 4951 generic.go:356] "Generic (PLEG): container finished" podID="4506abe9-6f43-4070-9b7b-64631f5be137" containerID="de7e9f9678a375bbdad9114fd00232caddcbc9c021d025f225dfc980640c68d3" exitCode=0 Oct 30 07:30:15.627268 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:15.627222 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" event={"ID":"4506abe9-6f43-4070-9b7b-64631f5be137","Type":"ContainerDied","Data":"de7e9f9678a375bbdad9114fd00232caddcbc9c021d025f225dfc980640c68d3"} Oct 30 07:30:15.627513 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:15.627275 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" Oct 30 07:30:15.627513 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:15.627312 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm" event={"ID":"4506abe9-6f43-4070-9b7b-64631f5be137","Type":"ContainerDied","Data":"cef59306f527fb871e9e0badce78cec3996e4a8c6e53ff09f516b9c7210529eb"} Oct 30 07:30:15.627513 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:15.627345 4951 scope.go:122] "RemoveContainer" containerID="de7e9f9678a375bbdad9114fd00232caddcbc9c021d025f225dfc980640c68d3" Oct 30 07:30:15.639746 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:15.639723 4951 scope.go:122] "RemoveContainer" containerID="a1054e6f9ac21300a17a382d2aaf5d3039c4ef2d281d401b5440b7fc91f77dca" Oct 30 07:30:15.650752 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:15.650726 4951 scope.go:122] "RemoveContainer" containerID="de7e9f9678a375bbdad9114fd00232caddcbc9c021d025f225dfc980640c68d3" Oct 30 07:30:15.651113 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:30:15.651088 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de7e9f9678a375bbdad9114fd00232caddcbc9c021d025f225dfc980640c68d3\": container with ID starting with de7e9f9678a375bbdad9114fd00232caddcbc9c021d025f225dfc980640c68d3 not found: ID does not exist" containerID="de7e9f9678a375bbdad9114fd00232caddcbc9c021d025f225dfc980640c68d3" Oct 30 07:30:15.651159 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:15.651123 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de7e9f9678a375bbdad9114fd00232caddcbc9c021d025f225dfc980640c68d3"} err="failed to get container status \"de7e9f9678a375bbdad9114fd00232caddcbc9c021d025f225dfc980640c68d3\": rpc error: code = NotFound desc = could not find container \"de7e9f9678a375bbdad9114fd00232caddcbc9c021d025f225dfc980640c68d3\": container with ID starting with de7e9f9678a375bbdad9114fd00232caddcbc9c021d025f225dfc980640c68d3 not found: ID does not exist" Oct 30 07:30:15.651159 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:15.651142 4951 scope.go:122] "RemoveContainer" containerID="a1054e6f9ac21300a17a382d2aaf5d3039c4ef2d281d401b5440b7fc91f77dca" Oct 30 07:30:15.651398 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:30:15.651379 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1054e6f9ac21300a17a382d2aaf5d3039c4ef2d281d401b5440b7fc91f77dca\": container with ID starting with a1054e6f9ac21300a17a382d2aaf5d3039c4ef2d281d401b5440b7fc91f77dca not found: ID does not exist" containerID="a1054e6f9ac21300a17a382d2aaf5d3039c4ef2d281d401b5440b7fc91f77dca" Oct 30 07:30:15.651432 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:15.651407 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1054e6f9ac21300a17a382d2aaf5d3039c4ef2d281d401b5440b7fc91f77dca"} err="failed to get container status \"a1054e6f9ac21300a17a382d2aaf5d3039c4ef2d281d401b5440b7fc91f77dca\": rpc error: code = NotFound desc = could not find container \"a1054e6f9ac21300a17a382d2aaf5d3039c4ef2d281d401b5440b7fc91f77dca\": container with ID starting with a1054e6f9ac21300a17a382d2aaf5d3039c4ef2d281d401b5440b7fc91f77dca not found: ID does not exist" Oct 30 07:30:15.699528 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:15.699496 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm"] Oct 30 07:30:15.714430 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:15.714380 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-5bb165-openshift-default-c7bffd59c-dwqzm"] Oct 30 07:30:15.764627 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:15.764556 4951 scope.go:122] "RemoveContainer" containerID="ff92c62b3cdeb7ed1210414bd31030e6e315083a2473fe52bfe3791f0631b7e1" Oct 30 07:30:15.773953 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:15.773929 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4506abe9-6f43-4070-9b7b-64631f5be137" path="/var/lib/kubelet/pods/4506abe9-6f43-4070-9b7b-64631f5be137/volumes" Oct 30 07:30:16.650888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:16.650854 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7" event={"ID":"2ab5010b-8726-4eb0-b3d7-55b1ba1be01b","Type":"ContainerStarted","Data":"4c6a09c114a20b308d325ead26632da21724ed1daba4ac8fefa1bcbbdfc5389d"} Oct 30 07:30:16.696189 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:16.696139 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7" podStartSLOduration=16.696126356 podStartE2EDuration="16.696126356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:16.694362948 +0000 UTC m=+2023.037410933" watchObservedRunningTime="2026-10-30 07:30:16.696126356 +0000 UTC m=+2023.039174365" Oct 30 07:30:16.962783 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:16.962744 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-46dac2-7d4f4449f5-864g5"] Oct 30 07:30:16.963391 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:16.963377 4951 cpu_manager.go:414] "containerMap: removing container" podUID="4506abe9-6f43-4070-9b7b-64631f5be137" containerName="istio-proxy" Oct 30 07:30:16.963455 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:16.963392 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4506abe9-6f43-4070-9b7b-64631f5be137" containerName="istio-proxy" Oct 30 07:30:16.963523 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:16.963514 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="4506abe9-6f43-4070-9b7b-64631f5be137" containerName="istio-proxy" Oct 30 07:30:16.967113 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:16.967095 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-46dac2-7d4f4449f5-864g5" Oct 30 07:30:16.989737 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:16.989711 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-46dac2\"" Oct 30 07:30:16.989901 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:16.989752 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-46dac2-dockercfg-nbgml\"" Oct 30 07:30:17.004303 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:16.998803 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-46dac2-7d4f4449f5-864g5"] Oct 30 07:30:17.063597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:17.062627 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-46dac2-7d4f4449f5-864g5\" (UID: \"6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-46dac2-7d4f4449f5-864g5" Oct 30 07:30:17.063597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:17.062854 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq4g6\" (UniqueName: \"kubernetes.io/projected/6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db-kube-api-access-fq4g6\") pod \"payload-pre-processing-e2e-autores-46dac2-7d4f4449f5-864g5\" (UID: \"6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-46dac2-7d4f4449f5-864g5" Oct 30 07:30:17.168595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:17.164487 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-46dac2-7d4f4449f5-864g5\" (UID: \"6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-46dac2-7d4f4449f5-864g5" Oct 30 07:30:17.168595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:17.164745 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq4g6\" (UniqueName: \"kubernetes.io/projected/6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db-kube-api-access-fq4g6\") pod \"payload-pre-processing-e2e-autores-46dac2-7d4f4449f5-864g5\" (UID: \"6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-46dac2-7d4f4449f5-864g5" Oct 30 07:30:17.168595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:17.166075 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-46dac2-7d4f4449f5-864g5\" (UID: \"6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-46dac2-7d4f4449f5-864g5" Oct 30 07:30:17.186647 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:17.182923 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-46dac2-6876b66bfb-7884m"] Oct 30 07:30:17.186647 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:17.183920 4951 cpu_manager.go:414] "containerMap: removing container" podUID="4506abe9-6f43-4070-9b7b-64631f5be137" containerName="istio-proxy" Oct 30 07:30:17.186647 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:17.183943 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4506abe9-6f43-4070-9b7b-64631f5be137" containerName="istio-proxy" Oct 30 07:30:17.186647 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:17.184141 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="4506abe9-6f43-4070-9b7b-64631f5be137" containerName="istio-proxy" Oct 30 07:30:17.210596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:17.191015 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-46dac2-6876b66bfb-7884m" Oct 30 07:30:17.294594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:17.285311 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ec6dc4f-18ea-4291-9da8-f95b4cee773d-plugins-config-volume\") pod \"payload-processing-e2e-autores-46dac2-6876b66bfb-7884m\" (UID: \"5ec6dc4f-18ea-4291-9da8-f95b4cee773d\") " pod="openshift-ingress/payload-processing-e2e-autores-46dac2-6876b66bfb-7884m" Oct 30 07:30:17.294594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:17.285528 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sprh6\" (UniqueName: \"kubernetes.io/projected/5ec6dc4f-18ea-4291-9da8-f95b4cee773d-kube-api-access-sprh6\") pod \"payload-processing-e2e-autores-46dac2-6876b66bfb-7884m\" (UID: \"5ec6dc4f-18ea-4291-9da8-f95b4cee773d\") " pod="openshift-ingress/payload-processing-e2e-autores-46dac2-6876b66bfb-7884m" Oct 30 07:30:17.369594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:17.362213 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-46dac2-6876b66bfb-7884m"] Oct 30 07:30:17.401708 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:17.398067 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sprh6\" (UniqueName: \"kubernetes.io/projected/5ec6dc4f-18ea-4291-9da8-f95b4cee773d-kube-api-access-sprh6\") pod \"payload-processing-e2e-autores-46dac2-6876b66bfb-7884m\" (UID: \"5ec6dc4f-18ea-4291-9da8-f95b4cee773d\") " pod="openshift-ingress/payload-processing-e2e-autores-46dac2-6876b66bfb-7884m" Oct 30 07:30:17.401708 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:17.398267 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ec6dc4f-18ea-4291-9da8-f95b4cee773d-plugins-config-volume\") pod \"payload-processing-e2e-autores-46dac2-6876b66bfb-7884m\" (UID: \"5ec6dc4f-18ea-4291-9da8-f95b4cee773d\") " pod="openshift-ingress/payload-processing-e2e-autores-46dac2-6876b66bfb-7884m" Oct 30 07:30:17.401708 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:17.399382 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ec6dc4f-18ea-4291-9da8-f95b4cee773d-plugins-config-volume\") pod \"payload-processing-e2e-autores-46dac2-6876b66bfb-7884m\" (UID: \"5ec6dc4f-18ea-4291-9da8-f95b4cee773d\") " pod="openshift-ingress/payload-processing-e2e-autores-46dac2-6876b66bfb-7884m" Oct 30 07:30:17.401708 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:17.399557 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq4g6\" (UniqueName: \"kubernetes.io/projected/6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db-kube-api-access-fq4g6\") pod \"payload-pre-processing-e2e-autores-46dac2-7d4f4449f5-864g5\" (UID: \"6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-46dac2-7d4f4449f5-864g5" Oct 30 07:30:17.425309 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:17.425279 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sprh6\" (UniqueName: \"kubernetes.io/projected/5ec6dc4f-18ea-4291-9da8-f95b4cee773d-kube-api-access-sprh6\") pod \"payload-processing-e2e-autores-46dac2-6876b66bfb-7884m\" (UID: \"5ec6dc4f-18ea-4291-9da8-f95b4cee773d\") " pod="openshift-ingress/payload-processing-e2e-autores-46dac2-6876b66bfb-7884m" Oct 30 07:30:17.606891 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:17.606854 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-46dac2-7d4f4449f5-864g5" Oct 30 07:30:17.661417 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:17.661382 4951 generic.go:356] "Generic (PLEG): container finished" podID="2ab5010b-8726-4eb0-b3d7-55b1ba1be01b" containerID="4c6a09c114a20b308d325ead26632da21724ed1daba4ac8fefa1bcbbdfc5389d" exitCode=60 Oct 30 07:30:17.661900 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:17.661423 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7" event={"ID":"2ab5010b-8726-4eb0-b3d7-55b1ba1be01b","Type":"ContainerDied","Data":"4c6a09c114a20b308d325ead26632da21724ed1daba4ac8fefa1bcbbdfc5389d"} Oct 30 07:30:17.661900 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:17.661470 4951 scope.go:122] "RemoveContainer" containerID="ff92c62b3cdeb7ed1210414bd31030e6e315083a2473fe52bfe3791f0631b7e1" Oct 30 07:30:17.662046 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:17.661910 4951 scope.go:122] "RemoveContainer" containerID="4c6a09c114a20b308d325ead26632da21724ed1daba4ac8fefa1bcbbdfc5389d" Oct 30 07:30:17.662297 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:30:17.662270 4951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7_odh-ai-gateway-infra(2ab5010b-8726-4eb0-b3d7-55b1ba1be01b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7" podUID="2ab5010b-8726-4eb0-b3d7-55b1ba1be01b" Oct 30 07:30:17.680168 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:17.678112 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-46dac2-6876b66bfb-7884m" Oct 30 07:30:17.705376 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:17.705348 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7"] Oct 30 07:30:17.918851 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:17.918823 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-46dac2-7d4f4449f5-864g5"] Oct 30 07:30:17.919181 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:30:17.919159 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c1d95b5_f9af_4d0b_b4b8_2d1af3b133db.slice/crio-3d5d9853a53ac4a1657df6c2aa6bbaaece79ba58070b18f9091a0b717d944db8 WatchSource:0}: Error finding container 3d5d9853a53ac4a1657df6c2aa6bbaaece79ba58070b18f9091a0b717d944db8: Status 404 returned error can't find the container with id 3d5d9853a53ac4a1657df6c2aa6bbaaece79ba58070b18f9091a0b717d944db8 Oct 30 07:30:18.015538 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:18.015488 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-46dac2-6876b66bfb-7884m"] Oct 30 07:30:18.018810 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:30:18.018772 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ec6dc4f_18ea_4291_9da8_f95b4cee773d.slice/crio-9753df2a0a64bd2bb10b8ab02d8e9c3bc426a2b7f1058dec3c957c17c76492bd WatchSource:0}: Error finding container 9753df2a0a64bd2bb10b8ab02d8e9c3bc426a2b7f1058dec3c957c17c76492bd: Status 404 returned error can't find the container with id 9753df2a0a64bd2bb10b8ab02d8e9c3bc426a2b7f1058dec3c957c17c76492bd Oct 30 07:30:18.671874 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:18.671837 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-46dac2-7d4f4449f5-864g5" event={"ID":"6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db","Type":"ContainerStarted","Data":"967ae142c7a09e1e00bb37f20e4dca7ffc310269dd16beb90796b1d5c9a45d7b"} Oct 30 07:30:18.672272 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:18.671909 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-46dac2-7d4f4449f5-864g5" Oct 30 07:30:18.672272 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:18.671935 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-46dac2-7d4f4449f5-864g5" event={"ID":"6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db","Type":"ContainerStarted","Data":"3d5d9853a53ac4a1657df6c2aa6bbaaece79ba58070b18f9091a0b717d944db8"} Oct 30 07:30:18.673767 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:18.673738 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-46dac2-6876b66bfb-7884m" event={"ID":"5ec6dc4f-18ea-4291-9da8-f95b4cee773d","Type":"ContainerStarted","Data":"c4dbe06fcc62acc8de52ff5ffdb6ee6b83cbab70043953982601f597e4899984"} Oct 30 07:30:18.673891 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:18.673792 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-46dac2-6876b66bfb-7884m" event={"ID":"5ec6dc4f-18ea-4291-9da8-f95b4cee773d","Type":"ContainerStarted","Data":"9753df2a0a64bd2bb10b8ab02d8e9c3bc426a2b7f1058dec3c957c17c76492bd"} Oct 30 07:30:18.673891 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:18.673825 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-46dac2-6876b66bfb-7884m" Oct 30 07:30:18.760193 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:18.760135 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-46dac2-7d4f4449f5-864g5" podStartSLOduration=2.760115461 podStartE2EDuration="2.760115461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:18.759719005 +0000 UTC m=+2025.102766971" watchObservedRunningTime="2026-10-30 07:30:18.760115461 +0000 UTC m=+2025.103163431" Oct 30 07:30:18.806317 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:18.806258 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-46dac2-6876b66bfb-7884m" podStartSLOduration=1.806241702 podStartE2EDuration="1.806241702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:18.802884897 +0000 UTC m=+2025.145932863" watchObservedRunningTime="2026-10-30 07:30:18.806241702 +0000 UTC m=+2025.149289666" Oct 30 07:30:18.825844 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:18.824173 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7" Oct 30 07:30:18.957067 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:18.955247 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ab5010b-8726-4eb0-b3d7-55b1ba1be01b-kube-api-access-k8v2s\" (UniqueName: \"kubernetes.io/projected/2ab5010b-8726-4eb0-b3d7-55b1ba1be01b-kube-api-access-k8v2s\") pod \"2ab5010b-8726-4eb0-b3d7-55b1ba1be01b\" (UID: \"2ab5010b-8726-4eb0-b3d7-55b1ba1be01b\") " Oct 30 07:30:18.957067 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:18.955419 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ab5010b-8726-4eb0-b3d7-55b1ba1be01b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ab5010b-8726-4eb0-b3d7-55b1ba1be01b-maas-api-service-ca\") pod \"2ab5010b-8726-4eb0-b3d7-55b1ba1be01b\" (UID: \"2ab5010b-8726-4eb0-b3d7-55b1ba1be01b\") " Oct 30 07:30:18.967582 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:18.967536 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ab5010b-8726-4eb0-b3d7-55b1ba1be01b-kube-api-access-k8v2s" pod "2ab5010b-8726-4eb0-b3d7-55b1ba1be01b" (UID: "2ab5010b-8726-4eb0-b3d7-55b1ba1be01b"). InnerVolumeSpecName "kube-api-access-k8v2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:30:18.977685 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:18.977628 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ab5010b-8726-4eb0-b3d7-55b1ba1be01b-maas-api-service-ca" pod "2ab5010b-8726-4eb0-b3d7-55b1ba1be01b" (UID: "2ab5010b-8726-4eb0-b3d7-55b1ba1be01b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:30:18.979389 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:18.978406 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8v2s\" (UniqueName: \"kubernetes.io/projected/2ab5010b-8726-4eb0-b3d7-55b1ba1be01b-kube-api-access-k8v2s\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:30:18.979389 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:18.978449 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ab5010b-8726-4eb0-b3d7-55b1ba1be01b-maas-api-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:30:19.626622 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:19.625727 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t" Oct 30 07:30:19.684220 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:19.684189 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7" Oct 30 07:30:19.684787 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:19.684219 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7" event={"ID":"2ab5010b-8726-4eb0-b3d7-55b1ba1be01b","Type":"ContainerDied","Data":"ad0900dca12ba94a641f49df6553656203b1e742e4713ead852ccc6239cf375b"} Oct 30 07:30:19.684787 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:19.684296 4951 scope.go:122] "RemoveContainer" containerID="4c6a09c114a20b308d325ead26632da21724ed1daba4ac8fefa1bcbbdfc5389d" Oct 30 07:30:19.744630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:19.744589 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7"] Oct 30 07:30:19.748991 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:19.748958 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889090-s5kq7"] Oct 30 07:30:19.773196 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:19.773159 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ab5010b-8726-4eb0-b3d7-55b1ba1be01b" path="/var/lib/kubelet/pods/2ab5010b-8726-4eb0-b3d7-55b1ba1be01b/volumes" Oct 30 07:30:23.024036 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:30:23.023999 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:30:25.779245 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:25.779208 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb"] Oct 30 07:30:25.780236 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:25.780211 4951 cpu_manager.go:414] "containerMap: removing container" podUID="2ab5010b-8726-4eb0-b3d7-55b1ba1be01b" containerName="cleanup" Oct 30 07:30:25.780425 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:25.780237 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ab5010b-8726-4eb0-b3d7-55b1ba1be01b" containerName="cleanup" Oct 30 07:30:25.780425 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:25.780288 4951 cpu_manager.go:414] "containerMap: removing container" podUID="2ab5010b-8726-4eb0-b3d7-55b1ba1be01b" containerName="cleanup" Oct 30 07:30:25.780425 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:25.780296 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ab5010b-8726-4eb0-b3d7-55b1ba1be01b" containerName="cleanup" Oct 30 07:30:25.780636 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:25.780453 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ab5010b-8726-4eb0-b3d7-55b1ba1be01b" containerName="cleanup" Oct 30 07:30:25.780636 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:25.780474 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ab5010b-8726-4eb0-b3d7-55b1ba1be01b" containerName="cleanup" Oct 30 07:30:25.786159 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:25.786135 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:30:25.811890 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:25.811864 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb"] Oct 30 07:30:25.820898 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:25.820875 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-d89593-openshift-default-dockercfg-frzxm\"" Oct 30 07:30:25.903390 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:25.900507 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istio-token\") pod \"e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:30:25.903390 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:25.900716 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istio-podinfo\") pod \"e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:30:25.903390 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:25.900795 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-workload-certs\") pod \"e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:30:25.903390 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:25.900830 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istiod-ca-cert\") pod \"e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:30:25.915596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:25.909769 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8wc9\" (UniqueName: \"kubernetes.io/projected/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-kube-api-access-k8wc9\") pod \"e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:30:25.915596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:25.909845 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istio-data\") pod \"e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:30:25.915596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:25.909909 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istio-envoy\") pod \"e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:30:25.915596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:25.909934 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-workload-socket\") pod \"e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:30:25.915596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:25.910113 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-credential-socket\") pod \"e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:30:26.012596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:26.012034 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istio-envoy\") pod \"e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:30:26.012596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:26.012121 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-workload-socket\") pod \"e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:30:26.012596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:26.012257 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-credential-socket\") pod \"e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:30:26.012596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:26.012366 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istio-token\") pod \"e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:30:26.012596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:26.012408 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istio-podinfo\") pod \"e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:30:26.012596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:26.012455 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-workload-certs\") pod \"e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:30:26.012596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:26.012479 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istiod-ca-cert\") pod \"e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:30:26.013120 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:26.012703 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8wc9\" (UniqueName: \"kubernetes.io/projected/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-kube-api-access-k8wc9\") pod \"e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:30:26.013120 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:26.012748 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istio-data\") pod \"e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:30:26.013235 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:26.013202 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istio-data\") pod \"e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:30:26.019158 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:26.018398 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istiod-ca-cert\") pod \"e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:30:26.019158 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:26.018708 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-workload-certs\") pod \"e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:30:26.019158 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:26.019273 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-credential-socket\") pod \"e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:30:26.019675 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:26.019495 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-workload-socket\") pod \"e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:30:26.043639 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:26.042067 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istio-podinfo\") pod \"e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:30:26.058595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:26.053211 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istio-envoy\") pod \"e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:30:26.065596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:26.063243 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8wc9\" (UniqueName: \"kubernetes.io/projected/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-kube-api-access-k8wc9\") pod \"e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:30:26.082260 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:26.078329 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istio-token\") pod \"e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:30:26.109601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:26.108858 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:30:26.540058 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:26.540026 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb"] Oct 30 07:30:26.540366 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:30:26.540345 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fe6f39a_7b8b_4e1b_aa0e_f25ba292e31b.slice/crio-c3df55d0f8975547f37c10419f043710583a1821726bf132a5f5c210a951f9f4 WatchSource:0}: Error finding container c3df55d0f8975547f37c10419f043710583a1821726bf132a5f5c210a951f9f4: Status 404 returned error can't find the container with id c3df55d0f8975547f37c10419f043710583a1821726bf132a5f5c210a951f9f4 Oct 30 07:30:26.543194 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:26.543160 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:30:26.543312 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:26.543225 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:30:26.543312 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:26.543256 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:30:26.763646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:26.763607 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" event={"ID":"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b","Type":"ContainerStarted","Data":"5feecbec4c9580ec0597e82af6032e5943d3900344ca4aeae190968dd7b027e6"} Oct 30 07:30:26.763951 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:26.763920 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" event={"ID":"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b","Type":"ContainerStarted","Data":"c3df55d0f8975547f37c10419f043710583a1821726bf132a5f5c210a951f9f4"} Oct 30 07:30:26.814537 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:26.814440 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" podStartSLOduration=1.814426611 podStartE2EDuration="1.814426611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:26.812893079 +0000 UTC m=+2033.155941043" watchObservedRunningTime="2026-10-30 07:30:26.814426611 +0000 UTC m=+2033.157474576" Oct 30 07:30:27.109644 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:27.109551 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:30:27.114629 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:27.114603 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:30:27.830436 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:27.830395 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:30:27.830983 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:27.830492 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:30:29.697594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:29.697157 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-46dac2-6876b66bfb-7884m" Oct 30 07:30:29.698095 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:29.697680 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-46dac2-7d4f4449f5-864g5" Oct 30 07:30:36.953602 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:36.952964 4951 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:30:36.954143 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:36.953112 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" podUID="e684243f-7d08-4ae2-b946-7cea93656322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:30:40.142649 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:40.142613 4951 generic.go:356] "Generic (PLEG): container finished" podID="e684243f-7d08-4ae2-b946-7cea93656322" containerID="dbc3d6ada8ab2566f28b0e170bc9eb549a31631e898acb0ea91e6de31d6e7aef" exitCode=0 Oct 30 07:30:40.143216 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:40.142693 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" event={"ID":"e684243f-7d08-4ae2-b946-7cea93656322","Type":"ContainerDied","Data":"dbc3d6ada8ab2566f28b0e170bc9eb549a31631e898acb0ea91e6de31d6e7aef"} Oct 30 07:30:40.143216 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:40.143143 4951 scope.go:122] "RemoveContainer" containerID="dbc3d6ada8ab2566f28b0e170bc9eb549a31631e898acb0ea91e6de31d6e7aef" Oct 30 07:30:40.145262 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:40.145224 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:30:40.145397 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:40.145333 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:30:40.145397 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:40.145384 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:30:41.157720 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:41.157679 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" event={"ID":"e684243f-7d08-4ae2-b946-7cea93656322","Type":"ContainerStarted","Data":"afcc967650aa7c9458537c5e2ccf39b40198c43f1836c91385a87512148ee5af"} Oct 30 07:30:41.462669 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:41.462557 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:30:41.462870 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:41.462854 4951 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 07:30:41.462939 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:41.462894 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" podUID="e684243f-7d08-4ae2-b946-7cea93656322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 07:30:42.463158 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:42.463117 4951 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 07:30:42.463713 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:42.463212 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" podUID="e684243f-7d08-4ae2-b946-7cea93656322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 07:30:43.463203 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:43.463162 4951 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 07:30:43.463628 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:43.463223 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" podUID="e684243f-7d08-4ae2-b946-7cea93656322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 07:30:44.463175 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:44.463134 4951 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 07:30:44.463371 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:44.463203 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" podUID="e684243f-7d08-4ae2-b946-7cea93656322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 07:30:45.463059 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:45.463016 4951 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 07:30:45.463381 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:45.463349 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" podUID="e684243f-7d08-4ae2-b946-7cea93656322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 07:30:46.463334 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:46.463294 4951 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 07:30:46.463557 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:46.463362 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" podUID="e684243f-7d08-4ae2-b946-7cea93656322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 07:30:47.613870 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:47.613832 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:30:47.614412 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:47.614310 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:30:47.615134 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:47.615112 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 07:30:53.057080 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:30:53.057029 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:30:58.327565 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:58.327524 4951 generic.go:356] "Generic (PLEG): container finished" podID="20ca6eb8-437b-4a4f-abea-d2ccf7243c8d" containerID="9bf5fc5978cdaa51c9de177715897074265891c9c8d5a374592115ff4b9e500b" exitCode=0 Oct 30 07:30:58.328068 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:58.327620 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" event={"ID":"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d","Type":"ContainerDied","Data":"9bf5fc5978cdaa51c9de177715897074265891c9c8d5a374592115ff4b9e500b"} Oct 30 07:30:58.328134 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:58.328116 4951 scope.go:122] "RemoveContainer" containerID="9bf5fc5978cdaa51c9de177715897074265891c9c8d5a374592115ff4b9e500b" Oct 30 07:30:58.330057 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:58.330019 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:30:58.330194 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:58.330114 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:30:58.330194 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:58.330168 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:30:59.397798 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:59.397755 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" event={"ID":"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d","Type":"ContainerStarted","Data":"add543c8fd49ba599894b1c13db7b04c3292a076a2e0b55d4b777430814951d2"} Oct 30 07:30:59.618270 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:59.618230 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:59.963393 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:59.963360 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:30:59.964183 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:59.964159 4951 patch_prober.go:29] interesting pod/e2e-autores-46dac2-openshift-default-5694949599-pr4k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 07:30:59.964269 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:30:59.964206 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" podUID="20ca6eb8-437b-4a4f-abea-d2ccf7243c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 07:31:00.969755 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:00.969708 4951 patch_prober.go:29] interesting pod/e2e-autores-46dac2-openshift-default-5694949599-pr4k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 07:31:00.970396 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:00.970352 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" podUID="20ca6eb8-437b-4a4f-abea-d2ccf7243c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 07:31:01.964202 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:01.964161 4951 patch_prober.go:29] interesting pod/e2e-autores-46dac2-openshift-default-5694949599-pr4k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 07:31:01.964389 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:01.964226 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" podUID="20ca6eb8-437b-4a4f-abea-d2ccf7243c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 07:31:02.968076 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:02.968041 4951 patch_prober.go:29] interesting pod/e2e-autores-46dac2-openshift-default-5694949599-pr4k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 07:31:02.968722 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:02.968681 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" podUID="20ca6eb8-437b-4a4f-abea-d2ccf7243c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 07:31:03.963794 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:03.963753 4951 patch_prober.go:29] interesting pod/e2e-autores-46dac2-openshift-default-5694949599-pr4k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 07:31:03.963980 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:03.963823 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" podUID="20ca6eb8-437b-4a4f-abea-d2ccf7243c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 07:31:04.062447 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.062413 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5"] Oct 30 07:31:04.063146 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.063130 4951 cpu_manager.go:414] "containerMap: removing container" podUID="2ab5010b-8726-4eb0-b3d7-55b1ba1be01b" containerName="cleanup" Oct 30 07:31:04.063192 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.063149 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ab5010b-8726-4eb0-b3d7-55b1ba1be01b" containerName="cleanup" Oct 30 07:31:04.063287 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.063277 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ab5010b-8726-4eb0-b3d7-55b1ba1be01b" containerName="cleanup" Oct 30 07:31:04.065912 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.065880 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 07:31:04.083049 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.083010 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-2e8b2a-openshift-default-dockercfg-r7jhj\"" Oct 30 07:31:04.102552 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.102485 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5"] Oct 30 07:31:04.156704 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.156417 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85d10cb3-066c-46fa-a149-209a92a409df-istiod-ca-cert\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 07:31:04.156704 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.156503 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-istio-envoy\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 07:31:04.156704 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.156636 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85d10cb3-066c-46fa-a149-209a92a409df-istio-token\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 07:31:04.157060 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.156811 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-workload-socket\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 07:31:04.157060 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.156851 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h5xl\" (UniqueName: \"kubernetes.io/projected/85d10cb3-066c-46fa-a149-209a92a409df-kube-api-access-2h5xl\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 07:31:04.157060 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.156954 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-workload-certs\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 07:31:04.157060 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.156985 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-credential-socket\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 07:31:04.157060 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.157025 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-istio-data\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 07:31:04.157060 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.157058 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85d10cb3-066c-46fa-a149-209a92a409df-istio-podinfo\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 07:31:04.258443 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.258395 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85d10cb3-066c-46fa-a149-209a92a409df-istio-token\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 07:31:04.258714 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.258548 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-workload-socket\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 07:31:04.258714 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.258599 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h5xl\" (UniqueName: \"kubernetes.io/projected/85d10cb3-066c-46fa-a149-209a92a409df-kube-api-access-2h5xl\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 07:31:04.258714 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.258693 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-workload-certs\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 07:31:04.258914 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.258717 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-credential-socket\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 07:31:04.258914 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.258743 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-istio-data\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 07:31:04.258914 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.258758 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85d10cb3-066c-46fa-a149-209a92a409df-istio-podinfo\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 07:31:04.258914 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.258787 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85d10cb3-066c-46fa-a149-209a92a409df-istiod-ca-cert\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 07:31:04.258914 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.258825 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-istio-envoy\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 07:31:04.259304 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.259273 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-credential-socket\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 07:31:04.259610 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.259584 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-istio-data\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 07:31:04.259704 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.259639 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-workload-certs\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 07:31:04.260093 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.260064 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85d10cb3-066c-46fa-a149-209a92a409df-istiod-ca-cert\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 07:31:04.260830 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.260605 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-workload-socket\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 07:31:04.262555 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.262527 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85d10cb3-066c-46fa-a149-209a92a409df-istio-podinfo\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 07:31:04.264120 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.264094 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-istio-envoy\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 07:31:04.284720 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.284692 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h5xl\" (UniqueName: \"kubernetes.io/projected/85d10cb3-066c-46fa-a149-209a92a409df-kube-api-access-2h5xl\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 07:31:04.284954 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.284938 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85d10cb3-066c-46fa-a149-209a92a409df-istio-token\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 07:31:04.379101 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.379063 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 07:31:04.758630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.758158 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:31:04.765641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.758273 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:31:04.765641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.760941 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:31:04.765641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.763799 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5"] Oct 30 07:31:04.968454 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.968372 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:31:04.969787 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:04.969763 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:31:05.556435 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:05.556384 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" event={"ID":"85d10cb3-066c-46fa-a149-209a92a409df","Type":"ContainerStarted","Data":"d539cd44faf50453ea041b22e793b75fd11a246981b674503412043d37ad6a60"} Oct 30 07:31:05.557027 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:05.556456 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" event={"ID":"85d10cb3-066c-46fa-a149-209a92a409df","Type":"ContainerStarted","Data":"fadfcd78c80bad9c9f09330d59e30a5c2b7fd6ed76c6e000cc3866362a68f4c2"} Oct 30 07:31:05.618741 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:05.618690 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" podStartSLOduration=1.6186716730000001 podStartE2EDuration="1.618671673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:31:05.617670739 +0000 UTC m=+2071.960718724" watchObservedRunningTime="2026-10-30 07:31:05.618671673 +0000 UTC m=+2071.961719641" Oct 30 07:31:06.384596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:06.383554 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 07:31:06.399597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:06.389002 4951 patch_prober.go:29] interesting pod/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 07:31:06.399597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:06.389079 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" podUID="85d10cb3-066c-46fa-a149-209a92a409df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 07:31:07.380127 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:07.380092 4951 patch_prober.go:29] interesting pod/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 07:31:07.380526 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:07.380150 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" podUID="85d10cb3-066c-46fa-a149-209a92a409df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 07:31:07.762315 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:07.762271 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v"] Oct 30 07:31:07.767252 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:07.767213 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:31:07.791263 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:07.791227 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v"] Oct 30 07:31:07.861994 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:07.861934 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:31:07.862243 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:07.862033 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:31:07.862243 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:07.862096 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:31:07.862243 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:07.862194 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:31:07.862243 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:07.862233 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:31:07.862604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:07.862279 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:31:07.862604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:07.862332 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ctlr\" (UniqueName: \"kubernetes.io/projected/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-kube-api-access-7ctlr\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:31:07.862604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:07.862364 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:31:07.862604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:07.862505 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:31:07.963956 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:07.963925 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:31:07.964134 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:07.963978 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:31:07.964134 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:07.964012 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:31:07.964134 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:07.964047 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ctlr\" (UniqueName: \"kubernetes.io/projected/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-kube-api-access-7ctlr\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:31:07.964134 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:07.964072 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:31:07.964134 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:07.964131 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:31:07.964327 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:07.964188 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:31:07.964327 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:07.964244 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:31:07.964327 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:07.964292 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:31:07.964999 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:07.964967 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:31:07.965154 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:07.965051 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:31:07.965353 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:07.965324 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:31:07.965687 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:07.965660 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:31:07.965923 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:07.965901 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:31:07.968084 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:07.968056 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:31:07.968254 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:07.968237 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:31:07.989556 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:07.989534 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:31:07.990216 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:07.990197 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ctlr\" (UniqueName: \"kubernetes.io/projected/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-kube-api-access-7ctlr\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:31:08.084411 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:08.084324 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:31:08.385612 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:08.384025 4951 patch_prober.go:29] interesting pod/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 07:31:08.385612 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:08.384071 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" podUID="85d10cb3-066c-46fa-a149-209a92a409df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 07:31:08.659607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:08.657065 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v"] Oct 30 07:31:08.660169 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:08.660128 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:31:08.660277 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:08.660196 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:31:08.660277 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:08.660228 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:31:09.380149 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:09.380108 4951 patch_prober.go:29] interesting pod/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 07:31:09.380344 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:09.380170 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" podUID="85d10cb3-066c-46fa-a149-209a92a409df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 07:31:09.588435 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:09.588396 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" event={"ID":"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5","Type":"ContainerStarted","Data":"6246129b09d8e1fb501eff895a819d035277c421b6cd4ca898a9c8962eab99a4"} Oct 30 07:31:09.588892 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:09.588453 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" event={"ID":"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5","Type":"ContainerStarted","Data":"583bb7c3b010a5be0e705cf0c674cddd199a6d76347df55bd1ed54ae78e3e403"} Oct 30 07:31:09.658653 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:09.656800 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" podStartSLOduration=2.656779675 podStartE2EDuration="2.656779675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:31:09.638622373 +0000 UTC m=+2075.981670340" watchObservedRunningTime="2026-10-30 07:31:09.656779675 +0000 UTC m=+2075.999827646" Oct 30 07:31:10.088338 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:10.088297 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:31:10.090594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:10.089256 4951 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 07:31:10.090594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:10.089318 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" podUID="f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 07:31:10.379659 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:10.379499 4951 patch_prober.go:29] interesting pod/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 07:31:10.379659 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:10.379558 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" podUID="85d10cb3-066c-46fa-a149-209a92a409df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 07:31:11.088761 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:11.088709 4951 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 07:31:11.089316 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:11.088798 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" podUID="f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 07:31:11.600654 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:11.597861 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 07:31:11.600654 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:11.598338 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 07:31:11.600654 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:11.600348 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 07:31:12.085508 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:12.085470 4951 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 07:31:12.085731 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:12.085530 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" podUID="f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 07:31:13.087019 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:13.086978 4951 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 07:31:13.087688 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:13.087648 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" podUID="f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 07:31:14.084907 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:14.084871 4951 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 07:31:14.085107 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:14.084927 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" podUID="f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 07:31:15.085079 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:15.085030 4951 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 07:31:15.085497 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:15.085093 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" podUID="f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 07:31:16.085741 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:16.085693 4951 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 07:31:16.086276 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:16.085794 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" podUID="f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 07:31:17.089925 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:17.089893 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:31:17.090422 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:17.090354 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:31:17.091250 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:17.091227 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:31:18.306377 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:18.306344 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn"] Oct 30 07:31:18.346648 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:18.346180 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn"] Oct 30 07:31:18.346648 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:18.346328 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:31:18.363936 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:18.363902 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-2761aa-openshift-default-dockercfg-pkcz8\"" Oct 30 07:31:18.475080 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:18.466947 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f44rk\" (UniqueName: \"kubernetes.io/projected/30bfffda-a440-4395-a471-3f302c3a0979-kube-api-access-f44rk\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:31:18.475452 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:18.475418 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30bfffda-a440-4395-a471-3f302c3a0979-istio-podinfo\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:31:18.475654 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:18.475634 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-istio-envoy\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:31:18.475780 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:18.475765 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-workload-certs\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:31:18.475969 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:18.475956 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-istio-data\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:31:18.476124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:18.476112 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-credential-socket\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:31:18.476256 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:18.476243 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30bfffda-a440-4395-a471-3f302c3a0979-istio-token\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:31:18.476464 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:18.476451 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-workload-socket\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:31:18.476634 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:18.476620 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30bfffda-a440-4395-a471-3f302c3a0979-istiod-ca-cert\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:31:18.582656 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:18.578540 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f44rk\" (UniqueName: \"kubernetes.io/projected/30bfffda-a440-4395-a471-3f302c3a0979-kube-api-access-f44rk\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:31:18.582909 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:18.582701 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30bfffda-a440-4395-a471-3f302c3a0979-istio-podinfo\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:31:18.582909 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:18.582798 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-istio-envoy\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:31:18.582909 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:18.582835 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-workload-certs\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:31:18.583103 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:18.582930 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-istio-data\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:31:18.583103 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:18.583016 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-credential-socket\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:31:18.583103 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:18.583081 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30bfffda-a440-4395-a471-3f302c3a0979-istio-token\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:31:18.583254 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:18.583224 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-workload-socket\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:31:18.583319 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:18.583301 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30bfffda-a440-4395-a471-3f302c3a0979-istiod-ca-cert\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:31:18.587309 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:18.583839 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-workload-certs\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:31:18.587309 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:18.584145 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30bfffda-a440-4395-a471-3f302c3a0979-istiod-ca-cert\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:31:18.587309 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:18.584365 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-istio-data\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:31:18.587309 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:18.584590 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-credential-socket\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:31:18.587309 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:18.585153 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-workload-socket\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:31:18.591202 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:18.591143 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-istio-envoy\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:31:18.591345 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:18.591250 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30bfffda-a440-4395-a471-3f302c3a0979-istio-podinfo\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:31:18.609699 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:18.609669 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30bfffda-a440-4395-a471-3f302c3a0979-istio-token\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:31:18.612322 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:18.612298 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f44rk\" (UniqueName: \"kubernetes.io/projected/30bfffda-a440-4395-a471-3f302c3a0979-kube-api-access-f44rk\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:31:18.744598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:18.743863 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:31:19.187737 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:19.187627 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn"] Oct 30 07:31:19.191287 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:31:19.191255 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30bfffda_a440_4395_a471_3f302c3a0979.slice/crio-04d0533bb4f34c4a3ce60bee605aba56981ef4744ff460a417eb0fca8d901946 WatchSource:0}: Error finding container 04d0533bb4f34c4a3ce60bee605aba56981ef4744ff460a417eb0fca8d901946: Status 404 returned error can't find the container with id 04d0533bb4f34c4a3ce60bee605aba56981ef4744ff460a417eb0fca8d901946 Oct 30 07:31:19.194393 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:19.194355 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:31:19.194471 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:19.194431 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:31:19.194471 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:19.194463 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:31:19.704473 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:19.704435 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" event={"ID":"30bfffda-a440-4395-a471-3f302c3a0979","Type":"ContainerStarted","Data":"0596e5544f9f4d86f8e8aa903963fe9a59257653f68f56341d7aeab037352898"} Oct 30 07:31:19.704977 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:19.704522 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" event={"ID":"30bfffda-a440-4395-a471-3f302c3a0979","Type":"ContainerStarted","Data":"04d0533bb4f34c4a3ce60bee605aba56981ef4744ff460a417eb0fca8d901946"} Oct 30 07:31:19.759597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:19.759099 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:31:20.748629 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:20.748596 4951 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 07:31:20.749238 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:20.749210 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" podUID="30bfffda-a440-4395-a471-3f302c3a0979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 07:31:21.758599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:21.758312 4951 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 07:31:21.758599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:21.758385 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" podUID="30bfffda-a440-4395-a471-3f302c3a0979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 07:31:22.768290 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:22.760678 4951 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 07:31:22.768290 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:22.760761 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" podUID="30bfffda-a440-4395-a471-3f302c3a0979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 07:31:23.186341 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:31:23.182308 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:31:23.760598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:23.746936 4951 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 07:31:23.760598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:23.747011 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" podUID="30bfffda-a440-4395-a471-3f302c3a0979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 07:31:24.753187 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:24.753138 4951 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 07:31:24.754257 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:24.753806 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" podUID="30bfffda-a440-4395-a471-3f302c3a0979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 07:31:25.747595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:25.746993 4951 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 07:31:25.747595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:25.747076 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" podUID="30bfffda-a440-4395-a471-3f302c3a0979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 07:31:26.745800 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:26.745755 4951 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 07:31:26.746307 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:26.745827 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" podUID="30bfffda-a440-4395-a471-3f302c3a0979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 07:31:27.646787 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:27.646675 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" podStartSLOduration=9.646634169 podStartE2EDuration="9.646634169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:31:19.759037652 +0000 UTC m=+2086.102085617" watchObservedRunningTime="2026-10-30 07:31:27.646634169 +0000 UTC m=+2093.989682127" Oct 30 07:31:27.652031 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:27.651997 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-054ef1-5b764f8cf-ms552"] Oct 30 07:31:27.659099 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:27.659073 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-054ef1-5b764f8cf-ms552" Oct 30 07:31:27.691055 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:27.691022 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-054ef1-5b764f8cf-ms552"] Oct 30 07:31:27.698870 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:27.698829 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-054ef1-dockercfg-9t5r8\"" Oct 30 07:31:27.707639 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:27.707021 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt58n\" (UniqueName: \"kubernetes.io/projected/070057c0-00c3-49dc-a4d2-868885c1685c-kube-api-access-dt58n\") pod \"payload-pre-processing-e2e-shared-b-w1-054ef1-5b764f8cf-ms552\" (UID: \"070057c0-00c3-49dc-a4d2-868885c1685c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-054ef1-5b764f8cf-ms552" Oct 30 07:31:27.707639 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:27.707412 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/070057c0-00c3-49dc-a4d2-868885c1685c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-054ef1-5b764f8cf-ms552\" (UID: \"070057c0-00c3-49dc-a4d2-868885c1685c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-054ef1-5b764f8cf-ms552" Oct 30 07:31:27.717849 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:27.717799 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-054ef1\"" Oct 30 07:31:27.754302 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:27.752735 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-054ef1-5dfd5878db-td8f2"] Oct 30 07:31:27.758008 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:27.757983 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:31:27.758343 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:27.758077 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-054ef1-5dfd5878db-td8f2" Oct 30 07:31:27.760034 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:27.760015 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:31:27.760964 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:27.760948 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:31:27.779925 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:27.779895 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-054ef1-5dfd5878db-td8f2"] Oct 30 07:31:27.821397 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:27.821352 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/070057c0-00c3-49dc-a4d2-868885c1685c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-054ef1-5b764f8cf-ms552\" (UID: \"070057c0-00c3-49dc-a4d2-868885c1685c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-054ef1-5b764f8cf-ms552" Oct 30 07:31:27.836631 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:27.831677 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt58n\" (UniqueName: \"kubernetes.io/projected/070057c0-00c3-49dc-a4d2-868885c1685c-kube-api-access-dt58n\") pod \"payload-pre-processing-e2e-shared-b-w1-054ef1-5b764f8cf-ms552\" (UID: \"070057c0-00c3-49dc-a4d2-868885c1685c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-054ef1-5b764f8cf-ms552" Oct 30 07:31:27.952347 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:27.952264 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjpsk\" (UniqueName: \"kubernetes.io/projected/8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc-kube-api-access-xjpsk\") pod \"payload-processing-e2e-shared-b-w1-054ef1-5dfd5878db-td8f2\" (UID: \"8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-054ef1-5dfd5878db-td8f2" Oct 30 07:31:27.952903 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:27.952876 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-054ef1-5dfd5878db-td8f2\" (UID: \"8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-054ef1-5dfd5878db-td8f2" Oct 30 07:31:27.953213 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:27.928775 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/070057c0-00c3-49dc-a4d2-868885c1685c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-054ef1-5b764f8cf-ms552\" (UID: \"070057c0-00c3-49dc-a4d2-868885c1685c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-054ef1-5b764f8cf-ms552" Oct 30 07:31:28.087013 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:28.068904 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-054ef1-5dfd5878db-td8f2\" (UID: \"8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-054ef1-5dfd5878db-td8f2" Oct 30 07:31:28.087013 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:28.069091 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjpsk\" (UniqueName: \"kubernetes.io/projected/8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc-kube-api-access-xjpsk\") pod \"payload-processing-e2e-shared-b-w1-054ef1-5dfd5878db-td8f2\" (UID: \"8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-054ef1-5dfd5878db-td8f2" Oct 30 07:31:28.088734 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:28.088621 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-054ef1-5dfd5878db-td8f2\" (UID: \"8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-054ef1-5dfd5878db-td8f2" Oct 30 07:31:28.186903 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:28.185536 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt58n\" (UniqueName: \"kubernetes.io/projected/070057c0-00c3-49dc-a4d2-868885c1685c-kube-api-access-dt58n\") pod \"payload-pre-processing-e2e-shared-b-w1-054ef1-5b764f8cf-ms552\" (UID: \"070057c0-00c3-49dc-a4d2-868885c1685c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-054ef1-5b764f8cf-ms552" Oct 30 07:31:28.189902 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:28.189875 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjpsk\" (UniqueName: \"kubernetes.io/projected/8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc-kube-api-access-xjpsk\") pod \"payload-processing-e2e-shared-b-w1-054ef1-5dfd5878db-td8f2\" (UID: \"8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-054ef1-5dfd5878db-td8f2" Oct 30 07:31:28.285431 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:28.285386 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-054ef1-5b764f8cf-ms552" Oct 30 07:31:28.389918 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:28.389861 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-054ef1-5dfd5878db-td8f2" Oct 30 07:31:28.625826 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:28.625535 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-054ef1-5b764f8cf-ms552"] Oct 30 07:31:28.626010 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:31:28.625945 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod070057c0_00c3_49dc_a4d2_868885c1685c.slice/crio-19ee519847e53e4ab27e8dc8ffdc9b003ca355ea3170857f8b0925d99fb50fb2 WatchSource:0}: Error finding container 19ee519847e53e4ab27e8dc8ffdc9b003ca355ea3170857f8b0925d99fb50fb2: Status 404 returned error can't find the container with id 19ee519847e53e4ab27e8dc8ffdc9b003ca355ea3170857f8b0925d99fb50fb2 Oct 30 07:31:28.762212 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:28.762182 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-054ef1-5dfd5878db-td8f2"] Oct 30 07:31:28.992725 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:28.992682 4951 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:31:28.992922 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:28.992727 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" podUID="3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:31:29.228633 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:29.228504 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-054ef1-5b764f8cf-ms552" event={"ID":"070057c0-00c3-49dc-a4d2-868885c1685c","Type":"ContainerStarted","Data":"ab330af512d31254a3ea895c288da180a4869d56a8d3976ef5d0698345ce2ea3"} Oct 30 07:31:29.228633 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:29.228595 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-054ef1-5b764f8cf-ms552" event={"ID":"070057c0-00c3-49dc-a4d2-868885c1685c","Type":"ContainerStarted","Data":"19ee519847e53e4ab27e8dc8ffdc9b003ca355ea3170857f8b0925d99fb50fb2"} Oct 30 07:31:29.230539 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:29.230479 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-054ef1-5b764f8cf-ms552" Oct 30 07:31:29.233613 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:29.233560 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-054ef1-5dfd5878db-td8f2" event={"ID":"8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc","Type":"ContainerStarted","Data":"12800c6ba21910303e830d18aef96448ad7d17b3f8e26849cd8adc165cde5467"} Oct 30 07:31:29.233756 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:29.233637 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-054ef1-5dfd5878db-td8f2" event={"ID":"8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc","Type":"ContainerStarted","Data":"d9f986e4b8067f1c201db3d6bd0b6efd678dc5d5848279a72d42765a0aae8b96"} Oct 30 07:31:29.233884 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:29.233858 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-054ef1-5dfd5878db-td8f2" Oct 30 07:31:29.275295 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:29.275235 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-054ef1-5b764f8cf-ms552" podStartSLOduration=2.275214999 podStartE2EDuration="2.275214999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:31:29.273920663 +0000 UTC m=+2095.616968654" watchObservedRunningTime="2026-10-30 07:31:29.275214999 +0000 UTC m=+2095.618262965" Oct 30 07:31:29.323454 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:29.323386 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-054ef1-5dfd5878db-td8f2" podStartSLOduration=2.323364881 podStartE2EDuration="2.323364881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:31:29.317237168 +0000 UTC m=+2095.660285134" watchObservedRunningTime="2026-10-30 07:31:29.323364881 +0000 UTC m=+2095.666412847" Oct 30 07:31:30.130007 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:30.129948 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd"] Oct 30 07:31:30.130591 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:30.130235 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" podUID="33872c4d-c6f0-4f12-9daf-4c23cb7b7d72" containerName="istio-proxy" containerID="cri-o://4c1c6de51f28ae2ae1f910f5b000f171e2e7a3baa1506d0564854093cd38cb24" gracePeriod=30 Oct 30 07:31:31.267817 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:31.267781 4951 generic.go:356] "Generic (PLEG): container finished" podID="db4df589-43fd-4413-8c51-de66dd1ec2e2" containerID="8398656a459109c55a08849f8661b4625027096103bbc31acefc58a5c6273516" exitCode=0 Oct 30 07:31:31.268365 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:31.267864 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" event={"ID":"db4df589-43fd-4413-8c51-de66dd1ec2e2","Type":"ContainerDied","Data":"8398656a459109c55a08849f8661b4625027096103bbc31acefc58a5c6273516"} Oct 30 07:31:31.268433 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:31.268361 4951 scope.go:122] "RemoveContainer" containerID="8398656a459109c55a08849f8661b4625027096103bbc31acefc58a5c6273516" Oct 30 07:31:31.269885 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:31.269859 4951 generic.go:356] "Generic (PLEG): container finished" podID="3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b" containerID="5feecbec4c9580ec0597e82af6032e5943d3900344ca4aeae190968dd7b027e6" exitCode=0 Oct 30 07:31:31.270003 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:31.269927 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" event={"ID":"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b","Type":"ContainerDied","Data":"5feecbec4c9580ec0597e82af6032e5943d3900344ca4aeae190968dd7b027e6"} Oct 30 07:31:31.270422 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:31.270408 4951 scope.go:122] "RemoveContainer" containerID="5feecbec4c9580ec0597e82af6032e5943d3900344ca4aeae190968dd7b027e6" Oct 30 07:31:31.270469 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:31.270412 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:31:31.270507 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:31.270492 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:31:31.270549 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:31.270535 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:31:31.271890 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:31.271861 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:31:31.271989 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:31.271937 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:31:31.272055 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:31.271991 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:31:32.322956 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:32.321722 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" event={"ID":"db4df589-43fd-4413-8c51-de66dd1ec2e2","Type":"ContainerStarted","Data":"d58458f3034fc77d477167f50303b7e144f2e9cc1028259f475a7df1baaf079c"} Oct 30 07:31:32.373453 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:32.373417 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" event={"ID":"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b","Type":"ContainerStarted","Data":"6afb17f54968c384715f464019e870ac9acb55cbcd700372d5fa74d706553fa7"} Oct 30 07:31:33.109751 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:33.109709 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:31:33.110149 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:33.110126 4951 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 07:31:33.110310 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:33.110289 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" podUID="3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 07:31:33.196335 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:33.196304 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:31:33.197522 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:33.197496 4951 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 07:31:33.197663 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:33.197544 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" podUID="db4df589-43fd-4413-8c51-de66dd1ec2e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 07:31:34.124604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:34.111535 4951 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 07:31:34.124604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:34.111624 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" podUID="3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 07:31:34.202595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:34.197078 4951 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 07:31:34.202595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:34.197135 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" podUID="db4df589-43fd-4413-8c51-de66dd1ec2e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 07:31:34.397036 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:34.396558 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/3.log" Oct 30 07:31:34.399803 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:34.399777 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/3.log" Oct 30 07:31:34.402937 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:34.402920 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 07:31:34.405542 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:34.405525 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 07:31:35.109501 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.109468 4951 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 07:31:35.109705 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.109518 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" podUID="3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 07:31:35.196230 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.196186 4951 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 07:31:35.196614 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.196253 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" podUID="db4df589-43fd-4413-8c51-de66dd1ec2e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 07:31:35.416056 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.414459 4951 generic.go:356] "Generic (PLEG): container finished" podID="33872c4d-c6f0-4f12-9daf-4c23cb7b7d72" containerID="4c1c6de51f28ae2ae1f910f5b000f171e2e7a3baa1506d0564854093cd38cb24" exitCode=0 Oct 30 07:31:35.416056 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.414655 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" event={"ID":"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72","Type":"ContainerDied","Data":"4c1c6de51f28ae2ae1f910f5b000f171e2e7a3baa1506d0564854093cd38cb24"} Oct 30 07:31:35.591314 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.591286 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:31:35.699607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.699476 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istio-data\") pod \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " Oct 30 07:31:35.699956 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.699931 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-workload-socket\") pod \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " Oct 30 07:31:35.701830 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.701801 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istio-envoy\") pod \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " Oct 30 07:31:35.702026 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.702004 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-workload-certs\") pod \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " Oct 30 07:31:35.702164 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.702147 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-credential-socket\") pod \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " Oct 30 07:31:35.702294 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.702278 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istio-podinfo\") pod \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " Oct 30 07:31:35.702422 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.702404 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-kube-api-access-4jtmn\" (UniqueName: \"kubernetes.io/projected/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-kube-api-access-4jtmn\") pod \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " Oct 30 07:31:35.702624 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.702606 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istio-token\" (UniqueName: \"kubernetes.io/projected/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istio-token\") pod \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " Oct 30 07:31:35.702778 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.702762 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istiod-ca-cert\") pod \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\" (UID: \"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72\") " Oct 30 07:31:35.704979 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.699951 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istio-data" pod "33872c4d-c6f0-4f12-9daf-4c23cb7b7d72" (UID: "33872c4d-c6f0-4f12-9daf-4c23cb7b7d72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:31:35.705106 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.700311 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-workload-socket" pod "33872c4d-c6f0-4f12-9daf-4c23cb7b7d72" (UID: "33872c4d-c6f0-4f12-9daf-4c23cb7b7d72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:31:35.705106 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.703358 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-workload-certs" pod "33872c4d-c6f0-4f12-9daf-4c23cb7b7d72" (UID: "33872c4d-c6f0-4f12-9daf-4c23cb7b7d72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:31:35.705106 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.703610 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-credential-socket" pod "33872c4d-c6f0-4f12-9daf-4c23cb7b7d72" (UID: "33872c4d-c6f0-4f12-9daf-4c23cb7b7d72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:31:35.706053 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.706028 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:31:35.706154 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.706059 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:31:35.706154 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.706074 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:31:35.706154 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.706089 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:31:35.706750 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.706716 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istiod-ca-cert" pod "33872c4d-c6f0-4f12-9daf-4c23cb7b7d72" (UID: "33872c4d-c6f0-4f12-9daf-4c23cb7b7d72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:31:35.720395 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.720350 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istio-token" pod "33872c4d-c6f0-4f12-9daf-4c23cb7b7d72" (UID: "33872c4d-c6f0-4f12-9daf-4c23cb7b7d72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:31:35.732940 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.731889 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istio-envoy" pod "33872c4d-c6f0-4f12-9daf-4c23cb7b7d72" (UID: "33872c4d-c6f0-4f12-9daf-4c23cb7b7d72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:31:35.740593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.739725 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istio-podinfo" pod "33872c4d-c6f0-4f12-9daf-4c23cb7b7d72" (UID: "33872c4d-c6f0-4f12-9daf-4c23cb7b7d72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:31:35.797550 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.797484 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-kube-api-access-4jtmn" pod "33872c4d-c6f0-4f12-9daf-4c23cb7b7d72" (UID: "33872c4d-c6f0-4f12-9daf-4c23cb7b7d72"). InnerVolumeSpecName "kube-api-access-4jtmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:31:35.809663 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.809619 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:31:35.809977 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.809957 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:31:35.810086 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.810072 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jtmn\" (UniqueName: \"kubernetes.io/projected/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-kube-api-access-4jtmn\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:31:35.810174 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.810163 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:31:35.810257 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.810246 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:31:35.906602 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:35.904442 4951 scope.go:122] "RemoveContainer" containerID="4c1c6de51f28ae2ae1f910f5b000f171e2e7a3baa1506d0564854093cd38cb24" Oct 30 07:31:36.110408 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:36.110370 4951 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 07:31:36.110598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:36.110429 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" podUID="3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 07:31:36.196016 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:36.195983 4951 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 07:31:36.196158 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:36.196036 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" podUID="db4df589-43fd-4413-8c51-de66dd1ec2e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 07:31:36.401220 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:36.401169 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:31:36.425596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:36.421563 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" Oct 30 07:31:36.425596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:36.421557 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd" event={"ID":"33872c4d-c6f0-4f12-9daf-4c23cb7b7d72","Type":"ContainerDied","Data":"03ebe115f09ead8e434920cbc974d5c68ed12d8da2c5f605b64de6ee96366522"} Oct 30 07:31:36.539604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:36.537364 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd"] Oct 30 07:31:36.557602 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:36.556190 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-thknd"] Oct 30 07:31:37.132597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:37.126327 4951 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 07:31:37.132597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:37.126392 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" podUID="3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 07:31:37.200621 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:37.197077 4951 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 07:31:37.200621 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:37.197137 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" podUID="db4df589-43fd-4413-8c51-de66dd1ec2e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 07:31:37.787032 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:37.786990 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33872c4d-c6f0-4f12-9daf-4c23cb7b7d72" path="/var/lib/kubelet/pods/33872c4d-c6f0-4f12-9daf-4c23cb7b7d72/volumes" Oct 30 07:31:38.114293 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:38.114208 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:31:38.114642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:38.114609 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:31:38.115682 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:38.115661 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 07:31:38.223518 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:38.223425 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:31:38.232048 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:38.232020 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 07:31:40.264079 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:40.264043 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-054ef1-5dfd5878db-td8f2" Oct 30 07:31:41.274284 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:41.274252 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-054ef1-5b764f8cf-ms552" Oct 30 07:31:49.576848 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:49.576770 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt"] Oct 30 07:31:49.577507 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:49.577492 4951 cpu_manager.go:414] "containerMap: removing container" podUID="33872c4d-c6f0-4f12-9daf-4c23cb7b7d72" containerName="istio-proxy" Oct 30 07:31:49.577564 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:49.577508 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33872c4d-c6f0-4f12-9daf-4c23cb7b7d72" containerName="istio-proxy" Oct 30 07:31:49.577646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:49.577635 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="33872c4d-c6f0-4f12-9daf-4c23cb7b7d72" containerName="istio-proxy" Oct 30 07:31:49.581124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:49.581107 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt" Oct 30 07:31:49.602675 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:49.602649 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt"] Oct 30 07:31:49.615620 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:49.612074 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-054ef1-serving-cert\"" Oct 30 07:31:49.664632 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:49.664592 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/147db729-cd3c-40f4-ae95-dd4db5869e60-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt\" (UID: \"147db729-cd3c-40f4-ae95-dd4db5869e60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt" Oct 30 07:31:49.664848 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:49.664674 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rzxd\" (UniqueName: \"kubernetes.io/projected/147db729-cd3c-40f4-ae95-dd4db5869e60-kube-api-access-2rzxd\") pod \"maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt\" (UID: \"147db729-cd3c-40f4-ae95-dd4db5869e60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt" Oct 30 07:31:49.664848 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:49.664724 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/147db729-cd3c-40f4-ae95-dd4db5869e60-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt\" (UID: \"147db729-cd3c-40f4-ae95-dd4db5869e60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt" Oct 30 07:31:49.664848 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:49.664812 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/147db729-cd3c-40f4-ae95-dd4db5869e60-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt\" (UID: \"147db729-cd3c-40f4-ae95-dd4db5869e60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt" Oct 30 07:31:49.774975 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:49.774119 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/147db729-cd3c-40f4-ae95-dd4db5869e60-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt\" (UID: \"147db729-cd3c-40f4-ae95-dd4db5869e60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt" Oct 30 07:31:49.775658 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:49.775627 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/147db729-cd3c-40f4-ae95-dd4db5869e60-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt\" (UID: \"147db729-cd3c-40f4-ae95-dd4db5869e60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt" Oct 30 07:31:49.776606 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:31:49.776040 4951 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-054ef1-serving-cert: secret "maas-api-e2e-shared-b-w1-054ef1-serving-cert" not found Oct 30 07:31:49.776895 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:31:49.776868 4951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/147db729-cd3c-40f4-ae95-dd4db5869e60-maas-api-tls podName:147db729-cd3c-40f4-ae95-dd4db5869e60 nodeName:}" failed. No retries permitted until 2026-10-30 07:31:50.276843811 +0000 UTC m=+2116.619891766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/147db729-cd3c-40f4-ae95-dd4db5869e60-maas-api-tls") pod "maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt" (UID: "147db729-cd3c-40f4-ae95-dd4db5869e60") : secret "maas-api-e2e-shared-b-w1-054ef1-serving-cert" not found Oct 30 07:31:49.777414 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:49.776127 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rzxd\" (UniqueName: \"kubernetes.io/projected/147db729-cd3c-40f4-ae95-dd4db5869e60-kube-api-access-2rzxd\") pod \"maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt\" (UID: \"147db729-cd3c-40f4-ae95-dd4db5869e60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt" Oct 30 07:31:49.777607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:49.777590 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/147db729-cd3c-40f4-ae95-dd4db5869e60-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt\" (UID: \"147db729-cd3c-40f4-ae95-dd4db5869e60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt" Oct 30 07:31:49.795599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:49.778625 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/147db729-cd3c-40f4-ae95-dd4db5869e60-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt\" (UID: \"147db729-cd3c-40f4-ae95-dd4db5869e60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt" Oct 30 07:31:49.830397 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:49.830307 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/147db729-cd3c-40f4-ae95-dd4db5869e60-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt\" (UID: \"147db729-cd3c-40f4-ae95-dd4db5869e60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt" Oct 30 07:31:49.862777 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:49.862730 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rzxd\" (UniqueName: \"kubernetes.io/projected/147db729-cd3c-40f4-ae95-dd4db5869e60-kube-api-access-2rzxd\") pod \"maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt\" (UID: \"147db729-cd3c-40f4-ae95-dd4db5869e60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt" Oct 30 07:31:50.329220 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:50.329186 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/147db729-cd3c-40f4-ae95-dd4db5869e60-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt\" (UID: \"147db729-cd3c-40f4-ae95-dd4db5869e60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt" Oct 30 07:31:50.331653 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:50.331626 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/147db729-cd3c-40f4-ae95-dd4db5869e60-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt\" (UID: \"147db729-cd3c-40f4-ae95-dd4db5869e60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt" Oct 30 07:31:50.496601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:50.495474 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt" Oct 30 07:31:50.900694 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:50.892208 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt"] Oct 30 07:31:50.905466 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:50.905435 4951 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 07:31:51.656265 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:51.656220 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt" event={"ID":"147db729-cd3c-40f4-ae95-dd4db5869e60","Type":"ContainerStarted","Data":"596c59fe2d3ef541b268b3701ac4ebf1cb14b9d259753c1d345f36cc1ee4ed21"} Oct 30 07:31:51.656485 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:51.656289 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt" event={"ID":"147db729-cd3c-40f4-ae95-dd4db5869e60","Type":"ContainerStarted","Data":"46fe1c49f1ebea1864124205389a22097568288524e29eec1cd1372097d8c4c7"} Oct 30 07:31:51.656485 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:51.656338 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt" Oct 30 07:31:51.705819 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:51.705761 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt" podStartSLOduration=2.432037253 podStartE2EDuration="2.705744459s" totalImagesPullingTime="273.707206ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:31:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:31:51.704061733 +0000 UTC m=+2118.047109719" watchObservedRunningTime="2026-10-30 07:31:51.705744459 +0000 UTC m=+2118.048792423" Oct 30 07:31:53.024106 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:31:53.024070 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:31:57.672456 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:31:57.672425 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt" Oct 30 07:32:10.681372 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:10.681338 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-46dac2-6876b66bfb-7884m"] Oct 30 07:32:10.715229 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:10.715196 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-46dac2-7d4f4449f5-864g5"] Oct 30 07:32:15.681666 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:15.681621 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-46dac2-6876b66bfb-7884m" podUID="5ec6dc4f-18ea-4291-9da8-f95b4cee773d" containerName="payload-processing" containerID="cri-o://c4dbe06fcc62acc8de52ff5ffdb6ee6b83cbab70043953982601f597e4899984" gracePeriod=25 Oct 30 07:32:15.716291 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:15.716256 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-46dac2-7d4f4449f5-864g5" podUID="6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db" containerName="payload-pre-processing" containerID="cri-o://967ae142c7a09e1e00bb37f20e4dca7ffc310269dd16beb90796b1d5c9a45d7b" gracePeriod=25 Oct 30 07:32:15.942925 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:15.942830 4951 generic.go:356] "Generic (PLEG): container finished" podID="6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db" containerID="967ae142c7a09e1e00bb37f20e4dca7ffc310269dd16beb90796b1d5c9a45d7b" exitCode=0 Oct 30 07:32:15.943192 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:15.942905 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-46dac2-7d4f4449f5-864g5" event={"ID":"6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db","Type":"ContainerDied","Data":"967ae142c7a09e1e00bb37f20e4dca7ffc310269dd16beb90796b1d5c9a45d7b"} Oct 30 07:32:15.944776 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:15.944750 4951 generic.go:356] "Generic (PLEG): container finished" podID="5ec6dc4f-18ea-4291-9da8-f95b4cee773d" containerID="c4dbe06fcc62acc8de52ff5ffdb6ee6b83cbab70043953982601f597e4899984" exitCode=0 Oct 30 07:32:15.944893 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:15.944877 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-46dac2-6876b66bfb-7884m" event={"ID":"5ec6dc4f-18ea-4291-9da8-f95b4cee773d","Type":"ContainerDied","Data":"c4dbe06fcc62acc8de52ff5ffdb6ee6b83cbab70043953982601f597e4899984"} Oct 30 07:32:16.017875 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:16.017502 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-46dac2-6876b66bfb-7884m" Oct 30 07:32:16.071474 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:16.071124 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ec6dc4f-18ea-4291-9da8-f95b4cee773d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ec6dc4f-18ea-4291-9da8-f95b4cee773d-plugins-config-volume\") pod \"5ec6dc4f-18ea-4291-9da8-f95b4cee773d\" (UID: \"5ec6dc4f-18ea-4291-9da8-f95b4cee773d\") " Oct 30 07:32:16.071474 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:16.071377 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ec6dc4f-18ea-4291-9da8-f95b4cee773d-kube-api-access-sprh6\" (UniqueName: \"kubernetes.io/projected/5ec6dc4f-18ea-4291-9da8-f95b4cee773d-kube-api-access-sprh6\") pod \"5ec6dc4f-18ea-4291-9da8-f95b4cee773d\" (UID: \"5ec6dc4f-18ea-4291-9da8-f95b4cee773d\") " Oct 30 07:32:16.072468 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:16.071807 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ec6dc4f-18ea-4291-9da8-f95b4cee773d-plugins-config-volume" pod "5ec6dc4f-18ea-4291-9da8-f95b4cee773d" (UID: "5ec6dc4f-18ea-4291-9da8-f95b4cee773d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:32:16.072468 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:16.072418 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ec6dc4f-18ea-4291-9da8-f95b4cee773d-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:32:16.074589 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:16.074554 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec6dc4f-18ea-4291-9da8-f95b4cee773d-kube-api-access-sprh6" pod "5ec6dc4f-18ea-4291-9da8-f95b4cee773d" (UID: "5ec6dc4f-18ea-4291-9da8-f95b4cee773d"). InnerVolumeSpecName "kube-api-access-sprh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:32:16.079360 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:16.079338 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-46dac2-7d4f4449f5-864g5" Oct 30 07:32:16.173374 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:16.173324 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db-kube-api-access-fq4g6\" (UniqueName: \"kubernetes.io/projected/6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db-kube-api-access-fq4g6\") pod \"6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db\" (UID: \"6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db\") " Oct 30 07:32:16.173744 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:16.173720 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db-plugins-config-volume\") pod \"6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db\" (UID: \"6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db\") " Oct 30 07:32:16.174340 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:16.174316 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sprh6\" (UniqueName: \"kubernetes.io/projected/5ec6dc4f-18ea-4291-9da8-f95b4cee773d-kube-api-access-sprh6\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:32:16.174768 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:16.174740 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db-plugins-config-volume" pod "6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db" (UID: "6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:32:16.177308 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:16.177274 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db-kube-api-access-fq4g6" pod "6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db" (UID: "6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db"). InnerVolumeSpecName "kube-api-access-fq4g6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:32:16.276080 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:16.276034 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:32:16.276080 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:16.276072 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq4g6\" (UniqueName: \"kubernetes.io/projected/6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db-kube-api-access-fq4g6\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:32:17.034596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:17.024478 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-46dac2-7d4f4449f5-864g5" event={"ID":"6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db","Type":"ContainerDied","Data":"3d5d9853a53ac4a1657df6c2aa6bbaaece79ba58070b18f9091a0b717d944db8"} Oct 30 07:32:17.034596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:17.024585 4951 scope.go:122] "RemoveContainer" containerID="967ae142c7a09e1e00bb37f20e4dca7ffc310269dd16beb90796b1d5c9a45d7b" Oct 30 07:32:17.034596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:17.024746 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-46dac2-7d4f4449f5-864g5" Oct 30 07:32:17.042584 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:17.042530 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-46dac2-6876b66bfb-7884m" event={"ID":"5ec6dc4f-18ea-4291-9da8-f95b4cee773d","Type":"ContainerDied","Data":"9753df2a0a64bd2bb10b8ab02d8e9c3bc426a2b7f1058dec3c957c17c76492bd"} Oct 30 07:32:17.043352 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:17.042904 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-46dac2-6876b66bfb-7884m" Oct 30 07:32:17.398777 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:17.398744 4951 scope.go:122] "RemoveContainer" containerID="c4dbe06fcc62acc8de52ff5ffdb6ee6b83cbab70043953982601f597e4899984" Oct 30 07:32:17.423088 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:17.423053 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-46dac2-7d4f4449f5-864g5"] Oct 30 07:32:17.431055 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:17.431018 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-46dac2-7d4f4449f5-864g5"] Oct 30 07:32:17.472629 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:17.472590 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-46dac2-6876b66bfb-7884m"] Oct 30 07:32:17.478494 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:17.478464 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-46dac2-6876b66bfb-7884m"] Oct 30 07:32:17.774287 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:17.774256 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ec6dc4f-18ea-4291-9da8-f95b4cee773d" path="/var/lib/kubelet/pods/5ec6dc4f-18ea-4291-9da8-f95b4cee773d/volumes" Oct 30 07:32:17.774747 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:17.774733 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db" path="/var/lib/kubelet/pods/6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db/volumes" Oct 30 07:32:23.025708 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:32:23.025669 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:32:25.062165 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:25.062122 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2"] Oct 30 07:32:25.062984 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:25.062921 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" podUID="20ca6eb8-437b-4a4f-abea-d2ccf7243c8d" containerName="istio-proxy" containerID="cri-o://add543c8fd49ba599894b1c13db7b04c3292a076a2e0b55d4b777430814951d2" gracePeriod=30 Oct 30 07:32:26.225952 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.225913 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn"] Oct 30 07:32:26.226955 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.226932 4951 cpu_manager.go:414] "containerMap: removing container" podUID="5ec6dc4f-18ea-4291-9da8-f95b4cee773d" containerName="payload-processing" Oct 30 07:32:26.226955 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.226956 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ec6dc4f-18ea-4291-9da8-f95b4cee773d" containerName="payload-processing" Oct 30 07:32:26.227161 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.227017 4951 cpu_manager.go:414] "containerMap: removing container" podUID="6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db" containerName="payload-pre-processing" Oct 30 07:32:26.227161 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.227026 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db" containerName="payload-pre-processing" Oct 30 07:32:26.227269 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.227190 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c1d95b5-f9af-4d0b-b4b8-2d1af3b133db" containerName="payload-pre-processing" Oct 30 07:32:26.227269 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.227210 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ec6dc4f-18ea-4291-9da8-f95b4cee773d" containerName="payload-processing" Oct 30 07:32:26.238799 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.238766 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:32:26.263858 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.261905 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-4fff82-openshift-default-dockercfg-45gk6\"" Oct 30 07:32:26.268944 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.268909 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn"] Oct 30 07:32:26.300072 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.300033 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfkjs\" (UniqueName: \"kubernetes.io/projected/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-kube-api-access-vfkjs\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:32:26.300289 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.300078 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istio-podinfo\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:32:26.300289 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.300175 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-workload-socket\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:32:26.300289 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.300228 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istiod-ca-cert\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:32:26.300501 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.300327 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istio-data\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:32:26.300501 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.300402 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-credential-socket\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:32:26.300616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.300505 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istio-envoy\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:32:26.300665 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.300654 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istio-token\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:32:26.300714 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.300688 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-workload-certs\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:32:26.402562 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.402516 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-credential-socket\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:32:26.402794 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.402644 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istio-envoy\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:32:26.402865 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.402810 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istio-token\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:32:26.402865 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.402859 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-workload-certs\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:32:26.402986 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.402913 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfkjs\" (UniqueName: \"kubernetes.io/projected/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-kube-api-access-vfkjs\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:32:26.402986 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.402950 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istio-podinfo\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:32:26.402986 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.402970 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-credential-socket\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:32:26.403152 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.402984 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-workload-socket\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:32:26.403152 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.403046 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istiod-ca-cert\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:32:26.403152 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.403080 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istio-data\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:32:26.403316 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.403203 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-workload-certs\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:32:26.403316 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.403261 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-workload-socket\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:32:26.403483 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.403458 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istio-data\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:32:26.403868 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.403842 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istiod-ca-cert\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:32:26.405155 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.405124 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istio-envoy\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:32:26.406103 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.406079 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istio-podinfo\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:32:26.428366 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.428334 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istio-token\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:32:26.428366 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.428353 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfkjs\" (UniqueName: \"kubernetes.io/projected/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-kube-api-access-vfkjs\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:32:26.575917 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:26.575878 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:32:27.033855 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:32:27.033826 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4fcbdd5_1641_44fe_9a29_0be0aceb6129.slice/crio-8609f0d16befb285494ebbc445e9a7e17b70228fdc88b210b27bc2cf9f6d421b WatchSource:0}: Error finding container 8609f0d16befb285494ebbc445e9a7e17b70228fdc88b210b27bc2cf9f6d421b: Status 404 returned error can't find the container with id 8609f0d16befb285494ebbc445e9a7e17b70228fdc88b210b27bc2cf9f6d421b Oct 30 07:32:27.034621 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:27.034556 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn"] Oct 30 07:32:27.036613 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:27.036551 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:32:27.036738 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:27.036642 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:32:27.036738 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:27.036686 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:32:27.145693 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:27.145654 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" event={"ID":"a4fcbdd5-1641-44fe-9a29-0be0aceb6129","Type":"ContainerStarted","Data":"5fb867544af00246958ab89e6516b37da0c80c9bc9badd3336b2b9f58173f21a"} Oct 30 07:32:27.145890 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:27.145722 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" event={"ID":"a4fcbdd5-1641-44fe-9a29-0be0aceb6129","Type":"ContainerStarted","Data":"8609f0d16befb285494ebbc445e9a7e17b70228fdc88b210b27bc2cf9f6d421b"} Oct 30 07:32:27.194055 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:27.193986 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" podStartSLOduration=1.193966239 podStartE2EDuration="1.193966239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:32:27.19348119 +0000 UTC m=+2153.536529181" watchObservedRunningTime="2026-10-30 07:32:27.193966239 +0000 UTC m=+2153.537014205" Oct 30 07:32:27.577563 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:27.577530 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:32:28.583912 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:28.583879 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:32:29.160524 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:29.160492 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:32:29.161623 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:29.161602 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:32:30.369677 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:30.369646 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:32:30.427306 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:30.427274 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-workload-socket\") pod \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " Oct 30 07:32:30.427493 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:30.427316 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istio-podinfo\") pod \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " Oct 30 07:32:30.427493 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:30.427347 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istio-envoy\") pod \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " Oct 30 07:32:30.427493 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:30.427384 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-credential-socket\") pod \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " Oct 30 07:32:30.427493 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:30.427458 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istio-data\") pod \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " Oct 30 07:32:30.427769 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:30.427499 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istio-token\" (UniqueName: \"kubernetes.io/projected/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istio-token\") pod \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " Oct 30 07:32:30.427769 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:30.427533 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istiod-ca-cert\") pod \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " Oct 30 07:32:30.427769 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:30.427603 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-workload-socket" pod "20ca6eb8-437b-4a4f-abea-d2ccf7243c8d" (UID: "20ca6eb8-437b-4a4f-abea-d2ccf7243c8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:32:30.427769 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:30.427610 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-kube-api-access-8r2xl\" (UniqueName: \"kubernetes.io/projected/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-kube-api-access-8r2xl\") pod \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " Oct 30 07:32:30.427769 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:30.427705 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-workload-certs\") pod \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\" (UID: \"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d\") " Oct 30 07:32:30.428008 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:30.427968 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istio-data" pod "20ca6eb8-437b-4a4f-abea-d2ccf7243c8d" (UID: "20ca6eb8-437b-4a4f-abea-d2ccf7243c8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:32:30.428008 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:30.427984 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-credential-socket" pod "20ca6eb8-437b-4a4f-abea-d2ccf7243c8d" (UID: "20ca6eb8-437b-4a4f-abea-d2ccf7243c8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:32:30.428217 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:30.428194 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-workload-certs" pod "20ca6eb8-437b-4a4f-abea-d2ccf7243c8d" (UID: "20ca6eb8-437b-4a4f-abea-d2ccf7243c8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:32:30.428320 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:30.428291 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istiod-ca-cert" pod "20ca6eb8-437b-4a4f-abea-d2ccf7243c8d" (UID: "20ca6eb8-437b-4a4f-abea-d2ccf7243c8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:32:30.428796 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:30.428773 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:32:30.428908 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:30.428800 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:32:30.428908 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:30.428812 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:32:30.428908 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:30.428821 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:32:30.428908 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:30.428829 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:32:30.430234 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:30.430210 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-kube-api-access-8r2xl" pod "20ca6eb8-437b-4a4f-abea-d2ccf7243c8d" (UID: "20ca6eb8-437b-4a4f-abea-d2ccf7243c8d"). InnerVolumeSpecName "kube-api-access-8r2xl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:32:30.430611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:30.430322 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istio-envoy" pod "20ca6eb8-437b-4a4f-abea-d2ccf7243c8d" (UID: "20ca6eb8-437b-4a4f-abea-d2ccf7243c8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:32:30.430611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:30.430340 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istio-token" pod "20ca6eb8-437b-4a4f-abea-d2ccf7243c8d" (UID: "20ca6eb8-437b-4a4f-abea-d2ccf7243c8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:32:30.430611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:30.430325 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istio-podinfo" pod "20ca6eb8-437b-4a4f-abea-d2ccf7243c8d" (UID: "20ca6eb8-437b-4a4f-abea-d2ccf7243c8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:32:30.530144 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:30.530113 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:32:30.530144 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:30.530140 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:32:30.530144 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:30.530148 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:32:30.530361 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:30.530159 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r2xl\" (UniqueName: \"kubernetes.io/projected/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d-kube-api-access-8r2xl\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:32:31.176972 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.176942 4951 generic.go:356] "Generic (PLEG): container finished" podID="20ca6eb8-437b-4a4f-abea-d2ccf7243c8d" containerID="add543c8fd49ba599894b1c13db7b04c3292a076a2e0b55d4b777430814951d2" exitCode=0 Oct 30 07:32:31.177144 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.176998 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" Oct 30 07:32:31.177144 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.177026 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" event={"ID":"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d","Type":"ContainerDied","Data":"add543c8fd49ba599894b1c13db7b04c3292a076a2e0b55d4b777430814951d2"} Oct 30 07:32:31.177144 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.177100 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2" event={"ID":"20ca6eb8-437b-4a4f-abea-d2ccf7243c8d","Type":"ContainerDied","Data":"653dd7c9dfecfb2de2aa30865ec3b508caae9b696be8f3cfaab281c9d43cd1ec"} Oct 30 07:32:31.177311 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.177147 4951 scope.go:122] "RemoveContainer" containerID="add543c8fd49ba599894b1c13db7b04c3292a076a2e0b55d4b777430814951d2" Oct 30 07:32:31.189313 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.189146 4951 scope.go:122] "RemoveContainer" containerID="9bf5fc5978cdaa51c9de177715897074265891c9c8d5a374592115ff4b9e500b" Oct 30 07:32:31.199733 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.199679 4951 scope.go:122] "RemoveContainer" containerID="add543c8fd49ba599894b1c13db7b04c3292a076a2e0b55d4b777430814951d2" Oct 30 07:32:31.200110 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:32:31.200089 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"add543c8fd49ba599894b1c13db7b04c3292a076a2e0b55d4b777430814951d2\": container with ID starting with add543c8fd49ba599894b1c13db7b04c3292a076a2e0b55d4b777430814951d2 not found: ID does not exist" containerID="add543c8fd49ba599894b1c13db7b04c3292a076a2e0b55d4b777430814951d2" Oct 30 07:32:31.200218 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.200126 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"add543c8fd49ba599894b1c13db7b04c3292a076a2e0b55d4b777430814951d2"} err="failed to get container status \"add543c8fd49ba599894b1c13db7b04c3292a076a2e0b55d4b777430814951d2\": rpc error: code = NotFound desc = could not find container \"add543c8fd49ba599894b1c13db7b04c3292a076a2e0b55d4b777430814951d2\": container with ID starting with add543c8fd49ba599894b1c13db7b04c3292a076a2e0b55d4b777430814951d2 not found: ID does not exist" Oct 30 07:32:31.200218 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.200156 4951 scope.go:122] "RemoveContainer" containerID="9bf5fc5978cdaa51c9de177715897074265891c9c8d5a374592115ff4b9e500b" Oct 30 07:32:31.200486 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:32:31.200464 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bf5fc5978cdaa51c9de177715897074265891c9c8d5a374592115ff4b9e500b\": container with ID starting with 9bf5fc5978cdaa51c9de177715897074265891c9c8d5a374592115ff4b9e500b not found: ID does not exist" containerID="9bf5fc5978cdaa51c9de177715897074265891c9c8d5a374592115ff4b9e500b" Oct 30 07:32:31.200657 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.200494 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bf5fc5978cdaa51c9de177715897074265891c9c8d5a374592115ff4b9e500b"} err="failed to get container status \"9bf5fc5978cdaa51c9de177715897074265891c9c8d5a374592115ff4b9e500b\": rpc error: code = NotFound desc = could not find container \"9bf5fc5978cdaa51c9de177715897074265891c9c8d5a374592115ff4b9e500b\": container with ID starting with 9bf5fc5978cdaa51c9de177715897074265891c9c8d5a374592115ff4b9e500b not found: ID does not exist" Oct 30 07:32:31.249286 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.249236 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2"] Oct 30 07:32:31.265300 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.265258 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-46dac2-openshift-default-5694949599-pr4k2"] Oct 30 07:32:31.680395 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.680360 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4fff82-5df4bf7875-mppll"] Oct 30 07:32:31.681037 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.681016 4951 cpu_manager.go:414] "containerMap: removing container" podUID="20ca6eb8-437b-4a4f-abea-d2ccf7243c8d" containerName="istio-proxy" Oct 30 07:32:31.681096 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.681039 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20ca6eb8-437b-4a4f-abea-d2ccf7243c8d" containerName="istio-proxy" Oct 30 07:32:31.681187 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.681175 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="20ca6eb8-437b-4a4f-abea-d2ccf7243c8d" containerName="istio-proxy" Oct 30 07:32:31.686218 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.686203 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4fff82-5df4bf7875-mppll" Oct 30 07:32:31.702757 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.702713 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-4fff82-dockercfg-wjjw5\"" Oct 30 07:32:31.702947 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.702840 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-4fff82\"" Oct 30 07:32:31.706152 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.705558 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4fff82-5df4bf7875-mppll"] Oct 30 07:32:31.713508 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.713475 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4fff82-6f8698f47d-bz92q"] Oct 30 07:32:31.714212 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.714190 4951 cpu_manager.go:414] "containerMap: removing container" podUID="20ca6eb8-437b-4a4f-abea-d2ccf7243c8d" containerName="istio-proxy" Oct 30 07:32:31.714212 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.714207 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20ca6eb8-437b-4a4f-abea-d2ccf7243c8d" containerName="istio-proxy" Oct 30 07:32:31.714405 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.714307 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="20ca6eb8-437b-4a4f-abea-d2ccf7243c8d" containerName="istio-proxy" Oct 30 07:32:31.717998 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.717971 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-4fff82-6f8698f47d-bz92q" Oct 30 07:32:31.771731 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.760075 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60fe230d-06d4-4c8a-ba90-71b87281216f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-4fff82-5df4bf7875-mppll\" (UID: \"60fe230d-06d4-4c8a-ba90-71b87281216f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4fff82-5df4bf7875-mppll" Oct 30 07:32:31.771731 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.760161 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr9m8\" (UniqueName: \"kubernetes.io/projected/c3de5bf3-0919-4e77-ab84-35ec6f218f88-kube-api-access-xr9m8\") pod \"payload-processing-e2e-explicit-4fff82-6f8698f47d-bz92q\" (UID: \"c3de5bf3-0919-4e77-ab84-35ec6f218f88\") " pod="openshift-ingress/payload-processing-e2e-explicit-4fff82-6f8698f47d-bz92q" Oct 30 07:32:31.771731 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.760195 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3de5bf3-0919-4e77-ab84-35ec6f218f88-plugins-config-volume\") pod \"payload-processing-e2e-explicit-4fff82-6f8698f47d-bz92q\" (UID: \"c3de5bf3-0919-4e77-ab84-35ec6f218f88\") " pod="openshift-ingress/payload-processing-e2e-explicit-4fff82-6f8698f47d-bz92q" Oct 30 07:32:31.771731 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.760263 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thgzh\" (UniqueName: \"kubernetes.io/projected/60fe230d-06d4-4c8a-ba90-71b87281216f-kube-api-access-thgzh\") pod \"payload-pre-processing-e2e-explicit-4fff82-5df4bf7875-mppll\" (UID: \"60fe230d-06d4-4c8a-ba90-71b87281216f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4fff82-5df4bf7875-mppll" Oct 30 07:32:31.984695 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.920594 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60fe230d-06d4-4c8a-ba90-71b87281216f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-4fff82-5df4bf7875-mppll\" (UID: \"60fe230d-06d4-4c8a-ba90-71b87281216f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4fff82-5df4bf7875-mppll" Oct 30 07:32:31.984695 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.920680 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr9m8\" (UniqueName: \"kubernetes.io/projected/c3de5bf3-0919-4e77-ab84-35ec6f218f88-kube-api-access-xr9m8\") pod \"payload-processing-e2e-explicit-4fff82-6f8698f47d-bz92q\" (UID: \"c3de5bf3-0919-4e77-ab84-35ec6f218f88\") " pod="openshift-ingress/payload-processing-e2e-explicit-4fff82-6f8698f47d-bz92q" Oct 30 07:32:31.984695 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.920718 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3de5bf3-0919-4e77-ab84-35ec6f218f88-plugins-config-volume\") pod \"payload-processing-e2e-explicit-4fff82-6f8698f47d-bz92q\" (UID: \"c3de5bf3-0919-4e77-ab84-35ec6f218f88\") " pod="openshift-ingress/payload-processing-e2e-explicit-4fff82-6f8698f47d-bz92q" Oct 30 07:32:31.984695 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.920789 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thgzh\" (UniqueName: \"kubernetes.io/projected/60fe230d-06d4-4c8a-ba90-71b87281216f-kube-api-access-thgzh\") pod \"payload-pre-processing-e2e-explicit-4fff82-5df4bf7875-mppll\" (UID: \"60fe230d-06d4-4c8a-ba90-71b87281216f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4fff82-5df4bf7875-mppll" Oct 30 07:32:31.984695 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.922091 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60fe230d-06d4-4c8a-ba90-71b87281216f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-4fff82-5df4bf7875-mppll\" (UID: \"60fe230d-06d4-4c8a-ba90-71b87281216f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4fff82-5df4bf7875-mppll" Oct 30 07:32:31.984695 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:31.935367 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3de5bf3-0919-4e77-ab84-35ec6f218f88-plugins-config-volume\") pod \"payload-processing-e2e-explicit-4fff82-6f8698f47d-bz92q\" (UID: \"c3de5bf3-0919-4e77-ab84-35ec6f218f88\") " pod="openshift-ingress/payload-processing-e2e-explicit-4fff82-6f8698f47d-bz92q" Oct 30 07:32:32.018608 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:32.009053 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thgzh\" (UniqueName: \"kubernetes.io/projected/60fe230d-06d4-4c8a-ba90-71b87281216f-kube-api-access-thgzh\") pod \"payload-pre-processing-e2e-explicit-4fff82-5df4bf7875-mppll\" (UID: \"60fe230d-06d4-4c8a-ba90-71b87281216f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4fff82-5df4bf7875-mppll" Oct 30 07:32:32.018608 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:32.018363 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20ca6eb8-437b-4a4f-abea-d2ccf7243c8d" path="/var/lib/kubelet/pods/20ca6eb8-437b-4a4f-abea-d2ccf7243c8d/volumes" Oct 30 07:32:32.018990 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:32.018903 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4fff82-6f8698f47d-bz92q"] Oct 30 07:32:32.063039 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:32.062991 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr9m8\" (UniqueName: \"kubernetes.io/projected/c3de5bf3-0919-4e77-ab84-35ec6f218f88-kube-api-access-xr9m8\") pod \"payload-processing-e2e-explicit-4fff82-6f8698f47d-bz92q\" (UID: \"c3de5bf3-0919-4e77-ab84-35ec6f218f88\") " pod="openshift-ingress/payload-processing-e2e-explicit-4fff82-6f8698f47d-bz92q" Oct 30 07:32:32.301396 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:32.301361 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4fff82-5df4bf7875-mppll" Oct 30 07:32:32.344502 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:32.344471 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-4fff82-6f8698f47d-bz92q" Oct 30 07:32:32.646563 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:32.646482 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4fff82-5df4bf7875-mppll"] Oct 30 07:32:32.664489 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:32:32.664461 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60fe230d_06d4_4c8a_ba90_71b87281216f.slice/crio-9d6a74e47ff917599ce35d2f46e526785e685e782bcf2369ea951f36c48ce9dc WatchSource:0}: Error finding container 9d6a74e47ff917599ce35d2f46e526785e685e782bcf2369ea951f36c48ce9dc: Status 404 returned error can't find the container with id 9d6a74e47ff917599ce35d2f46e526785e685e782bcf2369ea951f36c48ce9dc Oct 30 07:32:32.696101 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:32.696073 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4fff82-6f8698f47d-bz92q"] Oct 30 07:32:32.706956 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:32:32.706928 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3de5bf3_0919_4e77_ab84_35ec6f218f88.slice/crio-78437b29038391223536f1ec48cfc815d8ee31e2e94535403b39d25ff5d3250a WatchSource:0}: Error finding container 78437b29038391223536f1ec48cfc815d8ee31e2e94535403b39d25ff5d3250a: Status 404 returned error can't find the container with id 78437b29038391223536f1ec48cfc815d8ee31e2e94535403b39d25ff5d3250a Oct 30 07:32:33.206978 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:33.206930 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4fff82-5df4bf7875-mppll" event={"ID":"60fe230d-06d4-4c8a-ba90-71b87281216f","Type":"ContainerStarted","Data":"21490c7d9e2d7a9a91e914c1c44b3eb69e338beeb9fa90a3b2b9abe9d520fce2"} Oct 30 07:32:33.207226 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:33.207028 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4fff82-5df4bf7875-mppll" event={"ID":"60fe230d-06d4-4c8a-ba90-71b87281216f","Type":"ContainerStarted","Data":"9d6a74e47ff917599ce35d2f46e526785e685e782bcf2369ea951f36c48ce9dc"} Oct 30 07:32:33.208126 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:33.208101 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4fff82-5df4bf7875-mppll" Oct 30 07:32:33.210856 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:33.210366 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-4fff82-6f8698f47d-bz92q" event={"ID":"c3de5bf3-0919-4e77-ab84-35ec6f218f88","Type":"ContainerStarted","Data":"901bc329e0ce81e54d36383d87b6470ed51001549b92d990f7439b168cc41fe0"} Oct 30 07:32:33.210856 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:33.210416 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-4fff82-6f8698f47d-bz92q" event={"ID":"c3de5bf3-0919-4e77-ab84-35ec6f218f88","Type":"ContainerStarted","Data":"78437b29038391223536f1ec48cfc815d8ee31e2e94535403b39d25ff5d3250a"} Oct 30 07:32:33.211283 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:33.211262 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-4fff82-6f8698f47d-bz92q" Oct 30 07:32:33.253138 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:33.253057 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4fff82-5df4bf7875-mppll" podStartSLOduration=2.253014021 podStartE2EDuration="2.253014021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:32:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:32:33.25153312 +0000 UTC m=+2159.594581085" watchObservedRunningTime="2026-10-30 07:32:33.253014021 +0000 UTC m=+2159.596061992" Oct 30 07:32:33.299659 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:33.299597 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-4fff82-6f8698f47d-bz92q" podStartSLOduration=2.29956759 podStartE2EDuration="2.29956759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:32:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:32:33.29609173 +0000 UTC m=+2159.639139696" watchObservedRunningTime="2026-10-30 07:32:33.29956759 +0000 UTC m=+2159.642615555" Oct 30 07:32:45.229519 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:45.229488 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4fff82-5df4bf7875-mppll" Oct 30 07:32:45.230162 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:32:45.229609 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-4fff82-6f8698f47d-bz92q" Oct 30 07:32:53.027142 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:32:53.027098 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:33:23.030734 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:33:23.030688 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:33:23.744150 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:23.744107 4951 generic.go:356] "Generic (PLEG): container finished" podID="a4fcbdd5-1641-44fe-9a29-0be0aceb6129" containerID="5fb867544af00246958ab89e6516b37da0c80c9bc9badd3336b2b9f58173f21a" exitCode=0 Oct 30 07:33:23.744339 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:23.744158 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" event={"ID":"a4fcbdd5-1641-44fe-9a29-0be0aceb6129","Type":"ContainerDied","Data":"5fb867544af00246958ab89e6516b37da0c80c9bc9badd3336b2b9f58173f21a"} Oct 30 07:33:23.744673 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:23.744657 4951 scope.go:122] "RemoveContainer" containerID="5fb867544af00246958ab89e6516b37da0c80c9bc9badd3336b2b9f58173f21a" Oct 30 07:33:23.746357 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:23.746327 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:33:23.746464 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:23.746414 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:33:23.746532 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:23.746465 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:33:24.780656 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:24.780614 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" event={"ID":"a4fcbdd5-1641-44fe-9a29-0be0aceb6129","Type":"ContainerStarted","Data":"30a031e84d7757a8d50323e3faeef05357e5472d8358f52efb4aa6dd03d3860d"} Oct 30 07:33:25.577657 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:25.577610 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:33:25.579139 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:25.579107 4951 patch_prober.go:29] interesting pod/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 07:33:25.579303 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:25.579185 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" podUID="a4fcbdd5-1641-44fe-9a29-0be0aceb6129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 07:33:26.577857 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:26.577821 4951 patch_prober.go:29] interesting pod/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 07:33:26.578254 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:26.577917 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" podUID="a4fcbdd5-1641-44fe-9a29-0be0aceb6129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 07:33:27.576992 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:27.576943 4951 patch_prober.go:29] interesting pod/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 07:33:27.577217 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:27.577012 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" podUID="a4fcbdd5-1641-44fe-9a29-0be0aceb6129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 07:33:28.577038 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:28.576994 4951 patch_prober.go:29] interesting pod/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 07:33:28.577445 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:28.577068 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" podUID="a4fcbdd5-1641-44fe-9a29-0be0aceb6129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 07:33:29.160994 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:29.160956 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:33:29.577137 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:29.577098 4951 patch_prober.go:29] interesting pod/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 07:33:29.577704 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:29.577169 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" podUID="a4fcbdd5-1641-44fe-9a29-0be0aceb6129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 07:33:30.582098 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:30.582011 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:33:30.583046 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:30.583027 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:33:48.281261 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:48.281199 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8"] Oct 30 07:33:48.281752 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:48.281406 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" podUID="ea726207-9a8b-4494-bf06-c339b4114002" containerName="istio-proxy" containerID="cri-o://0ad8985d2330ab47dc9a23f7e96c28fbe1c7ca007df288947f58d25c90b4e078" gracePeriod=30 Oct 30 07:33:48.295542 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:48.295519 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4"] Oct 30 07:33:48.295807 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:48.295768 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" podUID="088ac46a-3235-420e-9680-74cb738f229c" containerName="istio-proxy" containerID="cri-o://5d6cc1f3ab5a51c8b8ccae89eaba690e43c909b9685a7033707cc3728f228a35" gracePeriod=30 Oct 30 07:33:48.299420 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:48.299399 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts"] Oct 30 07:33:48.299620 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:48.299588 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" podUID="246b0113-0e73-46b7-88f6-3fac16df57b5" containerName="istio-proxy" containerID="cri-o://7ec5f74bbbe481bfece2ca84efd2eb85dda0ae5eb67b577b439a8ea135d5dd6f" gracePeriod=30 Oct 30 07:33:48.319611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:48.311540 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp"] Oct 30 07:33:48.319611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:48.311733 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" podUID="da92d4f9-c801-4fc9-9eaf-589305c6851d" containerName="istio-proxy" containerID="cri-o://1be142c2868170592e4d7287912bc27beb9c90a4acf8158a8fe17496a75d9a83" gracePeriod=30 Oct 30 07:33:53.070841 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:33:53.070799 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:33:53.856645 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:53.854765 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:33:53.876040 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:53.876008 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:33:53.878928 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:53.878905 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:33:54.001400 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.000203 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/088ac46a-3235-420e-9680-74cb738f229c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/088ac46a-3235-420e-9680-74cb738f229c-istiod-ca-cert\") pod \"088ac46a-3235-420e-9680-74cb738f229c\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " Oct 30 07:33:54.001400 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.000960 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-workload-socket\") pod \"ea726207-9a8b-4494-bf06-c339b4114002\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " Oct 30 07:33:54.001400 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.001090 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea726207-9a8b-4494-bf06-c339b4114002-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea726207-9a8b-4494-bf06-c339b4114002-istiod-ca-cert\") pod \"ea726207-9a8b-4494-bf06-c339b4114002\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " Oct 30 07:33:54.001400 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.001154 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da92d4f9-c801-4fc9-9eaf-589305c6851d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da92d4f9-c801-4fc9-9eaf-589305c6851d-istio-podinfo\") pod \"da92d4f9-c801-4fc9-9eaf-589305c6851d\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " Oct 30 07:33:54.001400 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.001209 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-credential-socket\") pod \"ea726207-9a8b-4494-bf06-c339b4114002\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " Oct 30 07:33:54.001400 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.001268 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/088ac46a-3235-420e-9680-74cb738f229c-kube-api-access-bkxlk\" (UniqueName: \"kubernetes.io/projected/088ac46a-3235-420e-9680-74cb738f229c-kube-api-access-bkxlk\") pod \"088ac46a-3235-420e-9680-74cb738f229c\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " Oct 30 07:33:54.001400 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.001312 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-credential-socket\") pod \"da92d4f9-c801-4fc9-9eaf-589305c6851d\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " Oct 30 07:33:54.001400 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.001365 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-istio-data\") pod \"ea726207-9a8b-4494-bf06-c339b4114002\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " Oct 30 07:33:54.002050 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.001418 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-istio-data\") pod \"da92d4f9-c801-4fc9-9eaf-589305c6851d\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " Oct 30 07:33:54.002050 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.001445 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-workload-socket\") pod \"da92d4f9-c801-4fc9-9eaf-589305c6851d\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " Oct 30 07:33:54.002050 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.001491 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-credential-socket\") pod \"088ac46a-3235-420e-9680-74cb738f229c\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " Oct 30 07:33:54.002050 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.001522 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea726207-9a8b-4494-bf06-c339b4114002-kube-api-access-pftfx\" (UniqueName: \"kubernetes.io/projected/ea726207-9a8b-4494-bf06-c339b4114002-kube-api-access-pftfx\") pod \"ea726207-9a8b-4494-bf06-c339b4114002\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " Oct 30 07:33:54.002050 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.001628 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da92d4f9-c801-4fc9-9eaf-589305c6851d-istio-token\" (UniqueName: \"kubernetes.io/projected/da92d4f9-c801-4fc9-9eaf-589305c6851d-istio-token\") pod \"da92d4f9-c801-4fc9-9eaf-589305c6851d\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " Oct 30 07:33:54.002050 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.001679 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea726207-9a8b-4494-bf06-c339b4114002-istio-token\" (UniqueName: \"kubernetes.io/projected/ea726207-9a8b-4494-bf06-c339b4114002-istio-token\") pod \"ea726207-9a8b-4494-bf06-c339b4114002\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " Oct 30 07:33:54.002050 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.001730 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/088ac46a-3235-420e-9680-74cb738f229c-istio-token\" (UniqueName: \"kubernetes.io/projected/088ac46a-3235-420e-9680-74cb738f229c-istio-token\") pod \"088ac46a-3235-420e-9680-74cb738f229c\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " Oct 30 07:33:54.002050 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.001767 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-workload-certs\") pod \"da92d4f9-c801-4fc9-9eaf-589305c6851d\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " Oct 30 07:33:54.002050 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.001826 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-istio-envoy\") pod \"da92d4f9-c801-4fc9-9eaf-589305c6851d\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " Oct 30 07:33:54.002050 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.001909 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-istio-envoy\") pod \"ea726207-9a8b-4494-bf06-c339b4114002\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " Oct 30 07:33:54.002050 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.001979 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-istio-data\") pod \"088ac46a-3235-420e-9680-74cb738f229c\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " Oct 30 07:33:54.002050 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.002038 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-istio-envoy\") pod \"088ac46a-3235-420e-9680-74cb738f229c\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " Oct 30 07:33:54.002887 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.002077 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea726207-9a8b-4494-bf06-c339b4114002-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea726207-9a8b-4494-bf06-c339b4114002-istio-podinfo\") pod \"ea726207-9a8b-4494-bf06-c339b4114002\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " Oct 30 07:33:54.002887 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.002151 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-workload-socket\") pod \"088ac46a-3235-420e-9680-74cb738f229c\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " Oct 30 07:33:54.002887 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.002280 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da92d4f9-c801-4fc9-9eaf-589305c6851d-kube-api-access-jgzn4\" (UniqueName: \"kubernetes.io/projected/da92d4f9-c801-4fc9-9eaf-589305c6851d-kube-api-access-jgzn4\") pod \"da92d4f9-c801-4fc9-9eaf-589305c6851d\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " Oct 30 07:33:54.002887 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.002351 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-workload-certs\") pod \"ea726207-9a8b-4494-bf06-c339b4114002\" (UID: \"ea726207-9a8b-4494-bf06-c339b4114002\") " Oct 30 07:33:54.002887 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.002415 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/088ac46a-3235-420e-9680-74cb738f229c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/088ac46a-3235-420e-9680-74cb738f229c-istio-podinfo\") pod \"088ac46a-3235-420e-9680-74cb738f229c\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " Oct 30 07:33:54.002887 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.002686 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-workload-certs\") pod \"088ac46a-3235-420e-9680-74cb738f229c\" (UID: \"088ac46a-3235-420e-9680-74cb738f229c\") " Oct 30 07:33:54.002887 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.002762 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da92d4f9-c801-4fc9-9eaf-589305c6851d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da92d4f9-c801-4fc9-9eaf-589305c6851d-istiod-ca-cert\") pod \"da92d4f9-c801-4fc9-9eaf-589305c6851d\" (UID: \"da92d4f9-c801-4fc9-9eaf-589305c6851d\") " Oct 30 07:33:54.004087 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.004059 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da92d4f9-c801-4fc9-9eaf-589305c6851d-istiod-ca-cert" pod "da92d4f9-c801-4fc9-9eaf-589305c6851d" (UID: "da92d4f9-c801-4fc9-9eaf-589305c6851d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:33:54.004454 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.004433 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/088ac46a-3235-420e-9680-74cb738f229c-istiod-ca-cert" pod "088ac46a-3235-420e-9680-74cb738f229c" (UID: "088ac46a-3235-420e-9680-74cb738f229c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:33:54.004669 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.004649 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-workload-socket" pod "ea726207-9a8b-4494-bf06-c339b4114002" (UID: "ea726207-9a8b-4494-bf06-c339b4114002"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:33:54.006919 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.006887 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-workload-certs" pod "da92d4f9-c801-4fc9-9eaf-589305c6851d" (UID: "da92d4f9-c801-4fc9-9eaf-589305c6851d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:33:54.007230 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.007206 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-workload-certs" pod "ea726207-9a8b-4494-bf06-c339b4114002" (UID: "ea726207-9a8b-4494-bf06-c339b4114002"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:33:54.007437 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.007410 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-workload-socket" pod "088ac46a-3235-420e-9680-74cb738f229c" (UID: "088ac46a-3235-420e-9680-74cb738f229c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:33:54.008327 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.007767 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-istio-data" pod "ea726207-9a8b-4494-bf06-c339b4114002" (UID: "ea726207-9a8b-4494-bf06-c339b4114002"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:33:54.008327 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.008072 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea726207-9a8b-4494-bf06-c339b4114002-istiod-ca-cert" pod "ea726207-9a8b-4494-bf06-c339b4114002" (UID: "ea726207-9a8b-4494-bf06-c339b4114002"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:33:54.008327 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.008124 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/088ac46a-3235-420e-9680-74cb738f229c-istio-token" pod "088ac46a-3235-420e-9680-74cb738f229c" (UID: "088ac46a-3235-420e-9680-74cb738f229c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:33:54.008789 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.008763 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea726207-9a8b-4494-bf06-c339b4114002-istio-podinfo" pod "ea726207-9a8b-4494-bf06-c339b4114002" (UID: "ea726207-9a8b-4494-bf06-c339b4114002"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:33:54.009057 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.009024 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-istio-data" pod "088ac46a-3235-420e-9680-74cb738f229c" (UID: "088ac46a-3235-420e-9680-74cb738f229c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:33:54.009148 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.009053 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea726207-9a8b-4494-bf06-c339b4114002-istio-token" pod "ea726207-9a8b-4494-bf06-c339b4114002" (UID: "ea726207-9a8b-4494-bf06-c339b4114002"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:33:54.009256 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.009236 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-istio-data" pod "da92d4f9-c801-4fc9-9eaf-589305c6851d" (UID: "da92d4f9-c801-4fc9-9eaf-589305c6851d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:33:54.009409 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.009391 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-workload-socket" pod "da92d4f9-c801-4fc9-9eaf-589305c6851d" (UID: "da92d4f9-c801-4fc9-9eaf-589305c6851d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:33:54.009532 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.009519 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-credential-socket" pod "088ac46a-3235-420e-9680-74cb738f229c" (UID: "088ac46a-3235-420e-9680-74cb738f229c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:33:54.010012 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.009986 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da92d4f9-c801-4fc9-9eaf-589305c6851d-kube-api-access-jgzn4" pod "da92d4f9-c801-4fc9-9eaf-589305c6851d" (UID: "da92d4f9-c801-4fc9-9eaf-589305c6851d"). InnerVolumeSpecName "kube-api-access-jgzn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:33:54.010217 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.010197 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-workload-certs" pod "088ac46a-3235-420e-9680-74cb738f229c" (UID: "088ac46a-3235-420e-9680-74cb738f229c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:33:54.011488 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.011011 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-istio-envoy" pod "da92d4f9-c801-4fc9-9eaf-589305c6851d" (UID: "da92d4f9-c801-4fc9-9eaf-589305c6851d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:33:54.011488 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.011266 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-credential-socket" pod "da92d4f9-c801-4fc9-9eaf-589305c6851d" (UID: "da92d4f9-c801-4fc9-9eaf-589305c6851d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:33:54.011488 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.011452 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-credential-socket" pod "ea726207-9a8b-4494-bf06-c339b4114002" (UID: "ea726207-9a8b-4494-bf06-c339b4114002"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:33:54.013140 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.013107 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da92d4f9-c801-4fc9-9eaf-589305c6851d-istio-token" pod "da92d4f9-c801-4fc9-9eaf-589305c6851d" (UID: "da92d4f9-c801-4fc9-9eaf-589305c6851d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:33:54.016118 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.015721 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea726207-9a8b-4494-bf06-c339b4114002-kube-api-access-pftfx" pod "ea726207-9a8b-4494-bf06-c339b4114002" (UID: "ea726207-9a8b-4494-bf06-c339b4114002"). InnerVolumeSpecName "kube-api-access-pftfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:33:54.016118 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.015872 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da92d4f9-c801-4fc9-9eaf-589305c6851d-istio-podinfo" pod "da92d4f9-c801-4fc9-9eaf-589305c6851d" (UID: "da92d4f9-c801-4fc9-9eaf-589305c6851d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:33:54.016118 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.015909 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/088ac46a-3235-420e-9680-74cb738f229c-kube-api-access-bkxlk" pod "088ac46a-3235-420e-9680-74cb738f229c" (UID: "088ac46a-3235-420e-9680-74cb738f229c"). InnerVolumeSpecName "kube-api-access-bkxlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:33:54.017934 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.017912 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-istio-envoy" pod "ea726207-9a8b-4494-bf06-c339b4114002" (UID: "ea726207-9a8b-4494-bf06-c339b4114002"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:33:54.018036 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.018005 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-istio-envoy" pod "088ac46a-3235-420e-9680-74cb738f229c" (UID: "088ac46a-3235-420e-9680-74cb738f229c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:33:54.018763 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.018739 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/088ac46a-3235-420e-9680-74cb738f229c-istio-podinfo" pod "088ac46a-3235-420e-9680-74cb738f229c" (UID: "088ac46a-3235-420e-9680-74cb738f229c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:33:54.064683 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.064643 4951 generic.go:356] "Generic (PLEG): container finished" podID="088ac46a-3235-420e-9680-74cb738f229c" containerID="5d6cc1f3ab5a51c8b8ccae89eaba690e43c909b9685a7033707cc3728f228a35" exitCode=0 Oct 30 07:33:54.064835 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.064775 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" event={"ID":"088ac46a-3235-420e-9680-74cb738f229c","Type":"ContainerDied","Data":"5d6cc1f3ab5a51c8b8ccae89eaba690e43c909b9685a7033707cc3728f228a35"} Oct 30 07:33:54.064906 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.064865 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" event={"ID":"088ac46a-3235-420e-9680-74cb738f229c","Type":"ContainerDied","Data":"18b16e180063ed1e9ace0989030902816548352c860e7031b000da51942017cb"} Oct 30 07:33:54.064906 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.064896 4951 scope.go:122] "RemoveContainer" containerID="5d6cc1f3ab5a51c8b8ccae89eaba690e43c909b9685a7033707cc3728f228a35" Oct 30 07:33:54.065083 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.065064 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4" Oct 30 07:33:54.078141 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.078102 4951 generic.go:356] "Generic (PLEG): container finished" podID="246b0113-0e73-46b7-88f6-3fac16df57b5" containerID="7ec5f74bbbe481bfece2ca84efd2eb85dda0ae5eb67b577b439a8ea135d5dd6f" exitCode=0 Oct 30 07:33:54.078141 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.078127 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" event={"ID":"246b0113-0e73-46b7-88f6-3fac16df57b5","Type":"ContainerDied","Data":"7ec5f74bbbe481bfece2ca84efd2eb85dda0ae5eb67b577b439a8ea135d5dd6f"} Oct 30 07:33:54.080208 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.080163 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:33:54.081869 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.081065 4951 generic.go:356] "Generic (PLEG): container finished" podID="da92d4f9-c801-4fc9-9eaf-589305c6851d" containerID="1be142c2868170592e4d7287912bc27beb9c90a4acf8158a8fe17496a75d9a83" exitCode=0 Oct 30 07:33:54.081869 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.081113 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" Oct 30 07:33:54.081869 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.081264 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" event={"ID":"da92d4f9-c801-4fc9-9eaf-589305c6851d","Type":"ContainerDied","Data":"1be142c2868170592e4d7287912bc27beb9c90a4acf8158a8fe17496a75d9a83"} Oct 30 07:33:54.081869 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.081319 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp" event={"ID":"da92d4f9-c801-4fc9-9eaf-589305c6851d","Type":"ContainerDied","Data":"29ce253d5cf5127add75b837428b361a21323a392e3fd97d66631e9cb56101ee"} Oct 30 07:33:54.085173 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.085134 4951 generic.go:356] "Generic (PLEG): container finished" podID="ea726207-9a8b-4494-bf06-c339b4114002" containerID="0ad8985d2330ab47dc9a23f7e96c28fbe1c7ca007df288947f58d25c90b4e078" exitCode=0 Oct 30 07:33:54.085295 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.085208 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" Oct 30 07:33:54.085295 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.085216 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" event={"ID":"ea726207-9a8b-4494-bf06-c339b4114002","Type":"ContainerDied","Data":"0ad8985d2330ab47dc9a23f7e96c28fbe1c7ca007df288947f58d25c90b4e078"} Oct 30 07:33:54.085295 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.085273 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8" event={"ID":"ea726207-9a8b-4494-bf06-c339b4114002","Type":"ContainerDied","Data":"492535d95211181d56fc6842fc8c644838b702529ccad48c6d96eba41e75f616"} Oct 30 07:33:54.097557 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.097526 4951 scope.go:122] "RemoveContainer" containerID="5d6cc1f3ab5a51c8b8ccae89eaba690e43c909b9685a7033707cc3728f228a35" Oct 30 07:33:54.097978 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:33:54.097934 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d6cc1f3ab5a51c8b8ccae89eaba690e43c909b9685a7033707cc3728f228a35\": container with ID starting with 5d6cc1f3ab5a51c8b8ccae89eaba690e43c909b9685a7033707cc3728f228a35 not found: ID does not exist" containerID="5d6cc1f3ab5a51c8b8ccae89eaba690e43c909b9685a7033707cc3728f228a35" Oct 30 07:33:54.098132 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.097980 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d6cc1f3ab5a51c8b8ccae89eaba690e43c909b9685a7033707cc3728f228a35"} err="failed to get container status \"5d6cc1f3ab5a51c8b8ccae89eaba690e43c909b9685a7033707cc3728f228a35\": rpc error: code = NotFound desc = could not find container \"5d6cc1f3ab5a51c8b8ccae89eaba690e43c909b9685a7033707cc3728f228a35\": container with ID starting with 5d6cc1f3ab5a51c8b8ccae89eaba690e43c909b9685a7033707cc3728f228a35 not found: ID does not exist" Oct 30 07:33:54.098132 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.098011 4951 scope.go:122] "RemoveContainer" containerID="1be142c2868170592e4d7287912bc27beb9c90a4acf8158a8fe17496a75d9a83" Oct 30 07:33:54.105222 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.105130 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.105222 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.105161 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea726207-9a8b-4494-bf06-c339b4114002-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.105222 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.105175 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da92d4f9-c801-4fc9-9eaf-589305c6851d-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.105222 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.105188 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.105222 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.105201 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkxlk\" (UniqueName: \"kubernetes.io/projected/088ac46a-3235-420e-9680-74cb738f229c-kube-api-access-bkxlk\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.105222 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.105216 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.105679 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.105229 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.105679 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.105241 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.105679 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.105253 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.105679 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.105263 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.105679 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.105275 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pftfx\" (UniqueName: \"kubernetes.io/projected/ea726207-9a8b-4494-bf06-c339b4114002-kube-api-access-pftfx\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.105679 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.105287 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da92d4f9-c801-4fc9-9eaf-589305c6851d-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.105679 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.105298 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea726207-9a8b-4494-bf06-c339b4114002-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.105679 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.105310 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/088ac46a-3235-420e-9680-74cb738f229c-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.105679 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.105321 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.105679 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.105333 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da92d4f9-c801-4fc9-9eaf-589305c6851d-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.105679 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.105344 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.105679 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.105355 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.105679 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.105366 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.105679 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.105379 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea726207-9a8b-4494-bf06-c339b4114002-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.105679 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.105394 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.105679 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.105411 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgzn4\" (UniqueName: \"kubernetes.io/projected/da92d4f9-c801-4fc9-9eaf-589305c6851d-kube-api-access-jgzn4\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.105679 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.105425 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea726207-9a8b-4494-bf06-c339b4114002-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.105679 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.105440 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/088ac46a-3235-420e-9680-74cb738f229c-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.105679 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.105453 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/088ac46a-3235-420e-9680-74cb738f229c-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.105679 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.105466 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da92d4f9-c801-4fc9-9eaf-589305c6851d-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.105679 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.105479 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/088ac46a-3235-420e-9680-74cb738f229c-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.111148 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.111127 4951 scope.go:122] "RemoveContainer" containerID="1be142c2868170592e4d7287912bc27beb9c90a4acf8158a8fe17496a75d9a83" Oct 30 07:33:54.111503 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:33:54.111482 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1be142c2868170592e4d7287912bc27beb9c90a4acf8158a8fe17496a75d9a83\": container with ID starting with 1be142c2868170592e4d7287912bc27beb9c90a4acf8158a8fe17496a75d9a83 not found: ID does not exist" containerID="1be142c2868170592e4d7287912bc27beb9c90a4acf8158a8fe17496a75d9a83" Oct 30 07:33:54.111635 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.111514 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1be142c2868170592e4d7287912bc27beb9c90a4acf8158a8fe17496a75d9a83"} err="failed to get container status \"1be142c2868170592e4d7287912bc27beb9c90a4acf8158a8fe17496a75d9a83\": rpc error: code = NotFound desc = could not find container \"1be142c2868170592e4d7287912bc27beb9c90a4acf8158a8fe17496a75d9a83\": container with ID starting with 1be142c2868170592e4d7287912bc27beb9c90a4acf8158a8fe17496a75d9a83 not found: ID does not exist" Oct 30 07:33:54.111635 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.111533 4951 scope.go:122] "RemoveContainer" containerID="0ad8985d2330ab47dc9a23f7e96c28fbe1c7ca007df288947f58d25c90b4e078" Oct 30 07:33:54.121764 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.121554 4951 scope.go:122] "RemoveContainer" containerID="0ad8985d2330ab47dc9a23f7e96c28fbe1c7ca007df288947f58d25c90b4e078" Oct 30 07:33:54.122130 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:33:54.122100 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ad8985d2330ab47dc9a23f7e96c28fbe1c7ca007df288947f58d25c90b4e078\": container with ID starting with 0ad8985d2330ab47dc9a23f7e96c28fbe1c7ca007df288947f58d25c90b4e078 not found: ID does not exist" containerID="0ad8985d2330ab47dc9a23f7e96c28fbe1c7ca007df288947f58d25c90b4e078" Oct 30 07:33:54.122223 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.122142 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad8985d2330ab47dc9a23f7e96c28fbe1c7ca007df288947f58d25c90b4e078"} err="failed to get container status \"0ad8985d2330ab47dc9a23f7e96c28fbe1c7ca007df288947f58d25c90b4e078\": rpc error: code = NotFound desc = could not find container \"0ad8985d2330ab47dc9a23f7e96c28fbe1c7ca007df288947f58d25c90b4e078\": container with ID starting with 0ad8985d2330ab47dc9a23f7e96c28fbe1c7ca007df288947f58d25c90b4e078 not found: ID does not exist" Oct 30 07:33:54.148536 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.148507 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4"] Oct 30 07:33:54.166862 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.166830 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-q22w4"] Oct 30 07:33:54.206214 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.206180 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-istio-envoy\") pod \"246b0113-0e73-46b7-88f6-3fac16df57b5\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " Oct 30 07:33:54.206214 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.206219 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-credential-socket\") pod \"246b0113-0e73-46b7-88f6-3fac16df57b5\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " Oct 30 07:33:54.206467 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.206265 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-workload-socket\") pod \"246b0113-0e73-46b7-88f6-3fac16df57b5\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " Oct 30 07:33:54.206467 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.206290 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/246b0113-0e73-46b7-88f6-3fac16df57b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246b0113-0e73-46b7-88f6-3fac16df57b5-istiod-ca-cert\") pod \"246b0113-0e73-46b7-88f6-3fac16df57b5\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " Oct 30 07:33:54.206467 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.206325 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/246b0113-0e73-46b7-88f6-3fac16df57b5-kube-api-access-6wc6t\" (UniqueName: \"kubernetes.io/projected/246b0113-0e73-46b7-88f6-3fac16df57b5-kube-api-access-6wc6t\") pod \"246b0113-0e73-46b7-88f6-3fac16df57b5\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " Oct 30 07:33:54.206656 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.206531 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-workload-certs\") pod \"246b0113-0e73-46b7-88f6-3fac16df57b5\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " Oct 30 07:33:54.206656 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.206587 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-workload-socket" pod "246b0113-0e73-46b7-88f6-3fac16df57b5" (UID: "246b0113-0e73-46b7-88f6-3fac16df57b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:33:54.206656 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.206643 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/246b0113-0e73-46b7-88f6-3fac16df57b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246b0113-0e73-46b7-88f6-3fac16df57b5-istio-podinfo\") pod \"246b0113-0e73-46b7-88f6-3fac16df57b5\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " Oct 30 07:33:54.206818 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.206650 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-credential-socket" pod "246b0113-0e73-46b7-88f6-3fac16df57b5" (UID: "246b0113-0e73-46b7-88f6-3fac16df57b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:33:54.206818 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.206685 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/246b0113-0e73-46b7-88f6-3fac16df57b5-istio-token\" (UniqueName: \"kubernetes.io/projected/246b0113-0e73-46b7-88f6-3fac16df57b5-istio-token\") pod \"246b0113-0e73-46b7-88f6-3fac16df57b5\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " Oct 30 07:33:54.206818 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.206759 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-istio-data\") pod \"246b0113-0e73-46b7-88f6-3fac16df57b5\" (UID: \"246b0113-0e73-46b7-88f6-3fac16df57b5\") " Oct 30 07:33:54.206974 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.206866 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-workload-certs" pod "246b0113-0e73-46b7-88f6-3fac16df57b5" (UID: "246b0113-0e73-46b7-88f6-3fac16df57b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:33:54.207085 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.207065 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-istio-data" pod "246b0113-0e73-46b7-88f6-3fac16df57b5" (UID: "246b0113-0e73-46b7-88f6-3fac16df57b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:33:54.207360 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.207312 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.207453 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.207367 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.207453 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.207382 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.207453 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.207390 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.207697 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.207676 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/246b0113-0e73-46b7-88f6-3fac16df57b5-istiod-ca-cert" pod "246b0113-0e73-46b7-88f6-3fac16df57b5" (UID: "246b0113-0e73-46b7-88f6-3fac16df57b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:33:54.209282 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.209260 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-istio-envoy" pod "246b0113-0e73-46b7-88f6-3fac16df57b5" (UID: "246b0113-0e73-46b7-88f6-3fac16df57b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:33:54.209373 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.209337 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/246b0113-0e73-46b7-88f6-3fac16df57b5-istio-podinfo" pod "246b0113-0e73-46b7-88f6-3fac16df57b5" (UID: "246b0113-0e73-46b7-88f6-3fac16df57b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:33:54.209531 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.209514 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/246b0113-0e73-46b7-88f6-3fac16df57b5-kube-api-access-6wc6t" pod "246b0113-0e73-46b7-88f6-3fac16df57b5" (UID: "246b0113-0e73-46b7-88f6-3fac16df57b5"). InnerVolumeSpecName "kube-api-access-6wc6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:33:54.209826 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.209802 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/246b0113-0e73-46b7-88f6-3fac16df57b5-istio-token" pod "246b0113-0e73-46b7-88f6-3fac16df57b5" (UID: "246b0113-0e73-46b7-88f6-3fac16df57b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:33:54.259173 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.259131 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8"] Oct 30 07:33:54.274396 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.274347 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x82s8"] Oct 30 07:33:54.303051 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.303017 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp"] Oct 30 07:33:54.308927 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.308896 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246b0113-0e73-46b7-88f6-3fac16df57b5-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.308927 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.308929 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wc6t\" (UniqueName: \"kubernetes.io/projected/246b0113-0e73-46b7-88f6-3fac16df57b5-kube-api-access-6wc6t\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.309138 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.308945 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246b0113-0e73-46b7-88f6-3fac16df57b5-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.309138 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.308962 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246b0113-0e73-46b7-88f6-3fac16df57b5-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.309138 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.308975 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246b0113-0e73-46b7-88f6-3fac16df57b5-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:33:54.309483 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:54.309461 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-mgtfp"] Oct 30 07:33:55.094367 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:55.094338 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" Oct 30 07:33:55.094367 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:55.094364 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts" event={"ID":"246b0113-0e73-46b7-88f6-3fac16df57b5","Type":"ContainerDied","Data":"e50e88607dcf8b6c904dcc9ce151fa748f70c6625a42072308fbc014d6b5b53f"} Oct 30 07:33:55.094980 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:55.094431 4951 scope.go:122] "RemoveContainer" containerID="7ec5f74bbbe481bfece2ca84efd2eb85dda0ae5eb67b577b439a8ea135d5dd6f" Oct 30 07:33:55.155150 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:55.155117 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts"] Oct 30 07:33:55.167501 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:55.167469 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4r6ts"] Oct 30 07:33:55.773412 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:55.773374 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="088ac46a-3235-420e-9680-74cb738f229c" path="/var/lib/kubelet/pods/088ac46a-3235-420e-9680-74cb738f229c/volumes" Oct 30 07:33:55.773980 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:55.773963 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="246b0113-0e73-46b7-88f6-3fac16df57b5" path="/var/lib/kubelet/pods/246b0113-0e73-46b7-88f6-3fac16df57b5/volumes" Oct 30 07:33:55.774423 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:55.774410 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da92d4f9-c801-4fc9-9eaf-589305c6851d" path="/var/lib/kubelet/pods/da92d4f9-c801-4fc9-9eaf-589305c6851d/volumes" Oct 30 07:33:55.775251 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:33:55.775234 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea726207-9a8b-4494-bf06-c339b4114002" path="/var/lib/kubelet/pods/ea726207-9a8b-4494-bf06-c339b4114002/volumes" Oct 30 07:34:04.339808 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:04.339763 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4fff82-6f8698f47d-bz92q"] Oct 30 07:34:04.376129 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:04.376097 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4fff82-5df4bf7875-mppll"] Oct 30 07:34:09.340931 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:09.340887 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-4fff82-6f8698f47d-bz92q" podUID="c3de5bf3-0919-4e77-ab84-35ec6f218f88" containerName="payload-processing" containerID="cri-o://901bc329e0ce81e54d36383d87b6470ed51001549b92d990f7439b168cc41fe0" gracePeriod=25 Oct 30 07:34:09.376731 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:09.376667 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4fff82-5df4bf7875-mppll" podUID="60fe230d-06d4-4c8a-ba90-71b87281216f" containerName="payload-pre-processing" containerID="cri-o://21490c7d9e2d7a9a91e914c1c44b3eb69e338beeb9fa90a3b2b9abe9d520fce2" gracePeriod=25 Oct 30 07:34:09.681563 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:09.681535 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-4fff82-6f8698f47d-bz92q" Oct 30 07:34:09.750261 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:09.750221 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4fff82-5df4bf7875-mppll" Oct 30 07:34:09.873860 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:09.872199 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60fe230d-06d4-4c8a-ba90-71b87281216f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60fe230d-06d4-4c8a-ba90-71b87281216f-plugins-config-volume\") pod \"60fe230d-06d4-4c8a-ba90-71b87281216f\" (UID: \"60fe230d-06d4-4c8a-ba90-71b87281216f\") " Oct 30 07:34:09.873860 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:09.872286 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3de5bf3-0919-4e77-ab84-35ec6f218f88-kube-api-access-xr9m8\" (UniqueName: \"kubernetes.io/projected/c3de5bf3-0919-4e77-ab84-35ec6f218f88-kube-api-access-xr9m8\") pod \"c3de5bf3-0919-4e77-ab84-35ec6f218f88\" (UID: \"c3de5bf3-0919-4e77-ab84-35ec6f218f88\") " Oct 30 07:34:09.873860 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:09.872335 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3de5bf3-0919-4e77-ab84-35ec6f218f88-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3de5bf3-0919-4e77-ab84-35ec6f218f88-plugins-config-volume\") pod \"c3de5bf3-0919-4e77-ab84-35ec6f218f88\" (UID: \"c3de5bf3-0919-4e77-ab84-35ec6f218f88\") " Oct 30 07:34:09.873860 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:09.872436 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60fe230d-06d4-4c8a-ba90-71b87281216f-kube-api-access-thgzh\" (UniqueName: \"kubernetes.io/projected/60fe230d-06d4-4c8a-ba90-71b87281216f-kube-api-access-thgzh\") pod \"60fe230d-06d4-4c8a-ba90-71b87281216f\" (UID: \"60fe230d-06d4-4c8a-ba90-71b87281216f\") " Oct 30 07:34:09.891039 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:09.890946 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60fe230d-06d4-4c8a-ba90-71b87281216f-plugins-config-volume" pod "60fe230d-06d4-4c8a-ba90-71b87281216f" (UID: "60fe230d-06d4-4c8a-ba90-71b87281216f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:34:09.901023 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:09.900974 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3de5bf3-0919-4e77-ab84-35ec6f218f88-plugins-config-volume" pod "c3de5bf3-0919-4e77-ab84-35ec6f218f88" (UID: "c3de5bf3-0919-4e77-ab84-35ec6f218f88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:34:09.949888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:09.949820 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60fe230d-06d4-4c8a-ba90-71b87281216f-kube-api-access-thgzh" pod "60fe230d-06d4-4c8a-ba90-71b87281216f" (UID: "60fe230d-06d4-4c8a-ba90-71b87281216f"). InnerVolumeSpecName "kube-api-access-thgzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:34:09.952915 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:09.952876 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3de5bf3-0919-4e77-ab84-35ec6f218f88-kube-api-access-xr9m8" pod "c3de5bf3-0919-4e77-ab84-35ec6f218f88" (UID: "c3de5bf3-0919-4e77-ab84-35ec6f218f88"). InnerVolumeSpecName "kube-api-access-xr9m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:34:09.983087 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:09.980998 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thgzh\" (UniqueName: \"kubernetes.io/projected/60fe230d-06d4-4c8a-ba90-71b87281216f-kube-api-access-thgzh\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:34:09.983087 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:09.981048 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60fe230d-06d4-4c8a-ba90-71b87281216f-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:34:09.983087 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:09.981065 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr9m8\" (UniqueName: \"kubernetes.io/projected/c3de5bf3-0919-4e77-ab84-35ec6f218f88-kube-api-access-xr9m8\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:34:09.983087 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:09.981080 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3de5bf3-0919-4e77-ab84-35ec6f218f88-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:34:10.335180 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:10.335137 4951 generic.go:356] "Generic (PLEG): container finished" podID="60fe230d-06d4-4c8a-ba90-71b87281216f" containerID="21490c7d9e2d7a9a91e914c1c44b3eb69e338beeb9fa90a3b2b9abe9d520fce2" exitCode=0 Oct 30 07:34:10.335378 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:10.335188 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4fff82-5df4bf7875-mppll" Oct 30 07:34:10.335378 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:10.335199 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4fff82-5df4bf7875-mppll" event={"ID":"60fe230d-06d4-4c8a-ba90-71b87281216f","Type":"ContainerDied","Data":"21490c7d9e2d7a9a91e914c1c44b3eb69e338beeb9fa90a3b2b9abe9d520fce2"} Oct 30 07:34:10.335378 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:10.335257 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4fff82-5df4bf7875-mppll" event={"ID":"60fe230d-06d4-4c8a-ba90-71b87281216f","Type":"ContainerDied","Data":"9d6a74e47ff917599ce35d2f46e526785e685e782bcf2369ea951f36c48ce9dc"} Oct 30 07:34:10.335378 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:10.335290 4951 scope.go:122] "RemoveContainer" containerID="21490c7d9e2d7a9a91e914c1c44b3eb69e338beeb9fa90a3b2b9abe9d520fce2" Oct 30 07:34:10.336883 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:10.336848 4951 generic.go:356] "Generic (PLEG): container finished" podID="c3de5bf3-0919-4e77-ab84-35ec6f218f88" containerID="901bc329e0ce81e54d36383d87b6470ed51001549b92d990f7439b168cc41fe0" exitCode=0 Oct 30 07:34:10.337039 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:10.336894 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-4fff82-6f8698f47d-bz92q" Oct 30 07:34:10.337039 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:10.336927 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-4fff82-6f8698f47d-bz92q" event={"ID":"c3de5bf3-0919-4e77-ab84-35ec6f218f88","Type":"ContainerDied","Data":"901bc329e0ce81e54d36383d87b6470ed51001549b92d990f7439b168cc41fe0"} Oct 30 07:34:10.337039 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:10.336994 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-4fff82-6f8698f47d-bz92q" event={"ID":"c3de5bf3-0919-4e77-ab84-35ec6f218f88","Type":"ContainerDied","Data":"78437b29038391223536f1ec48cfc815d8ee31e2e94535403b39d25ff5d3250a"} Oct 30 07:34:10.373148 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:10.369139 4951 scope.go:122] "RemoveContainer" containerID="21490c7d9e2d7a9a91e914c1c44b3eb69e338beeb9fa90a3b2b9abe9d520fce2" Oct 30 07:34:10.389959 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:34:10.389020 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21490c7d9e2d7a9a91e914c1c44b3eb69e338beeb9fa90a3b2b9abe9d520fce2\": container with ID starting with 21490c7d9e2d7a9a91e914c1c44b3eb69e338beeb9fa90a3b2b9abe9d520fce2 not found: ID does not exist" containerID="21490c7d9e2d7a9a91e914c1c44b3eb69e338beeb9fa90a3b2b9abe9d520fce2" Oct 30 07:34:10.389959 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:10.389072 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21490c7d9e2d7a9a91e914c1c44b3eb69e338beeb9fa90a3b2b9abe9d520fce2"} err="failed to get container status \"21490c7d9e2d7a9a91e914c1c44b3eb69e338beeb9fa90a3b2b9abe9d520fce2\": rpc error: code = NotFound desc = could not find container \"21490c7d9e2d7a9a91e914c1c44b3eb69e338beeb9fa90a3b2b9abe9d520fce2\": container with ID starting with 21490c7d9e2d7a9a91e914c1c44b3eb69e338beeb9fa90a3b2b9abe9d520fce2 not found: ID does not exist" Oct 30 07:34:10.389959 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:10.389098 4951 scope.go:122] "RemoveContainer" containerID="901bc329e0ce81e54d36383d87b6470ed51001549b92d990f7439b168cc41fe0" Oct 30 07:34:10.403234 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:10.403208 4951 scope.go:122] "RemoveContainer" containerID="901bc329e0ce81e54d36383d87b6470ed51001549b92d990f7439b168cc41fe0" Oct 30 07:34:10.403689 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:34:10.403563 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"901bc329e0ce81e54d36383d87b6470ed51001549b92d990f7439b168cc41fe0\": container with ID starting with 901bc329e0ce81e54d36383d87b6470ed51001549b92d990f7439b168cc41fe0 not found: ID does not exist" containerID="901bc329e0ce81e54d36383d87b6470ed51001549b92d990f7439b168cc41fe0" Oct 30 07:34:10.403934 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:10.403694 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"901bc329e0ce81e54d36383d87b6470ed51001549b92d990f7439b168cc41fe0"} err="failed to get container status \"901bc329e0ce81e54d36383d87b6470ed51001549b92d990f7439b168cc41fe0\": rpc error: code = NotFound desc = could not find container \"901bc329e0ce81e54d36383d87b6470ed51001549b92d990f7439b168cc41fe0\": container with ID starting with 901bc329e0ce81e54d36383d87b6470ed51001549b92d990f7439b168cc41fe0 not found: ID does not exist" Oct 30 07:34:10.442680 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:10.442648 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4fff82-5df4bf7875-mppll"] Oct 30 07:34:10.456071 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:10.456044 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4fff82-5df4bf7875-mppll"] Oct 30 07:34:10.527455 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:10.524612 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4fff82-6f8698f47d-bz92q"] Oct 30 07:34:10.538619 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:10.538259 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4fff82-6f8698f47d-bz92q"] Oct 30 07:34:11.047119 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:11.047084 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54"] Oct 30 07:34:11.047339 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:11.047318 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" podUID="2ea0e265-4a3b-46ff-98d1-129054c698df" containerName="istio-proxy" containerID="cri-o://a75ce083500fd2e5fa40312feaef96567507cceab1f54a5781c4db842454b370" gracePeriod=30 Oct 30 07:34:11.773251 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:11.773220 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60fe230d-06d4-4c8a-ba90-71b87281216f" path="/var/lib/kubelet/pods/60fe230d-06d4-4c8a-ba90-71b87281216f/volumes" Oct 30 07:34:11.773737 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:11.773719 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3de5bf3-0919-4e77-ab84-35ec6f218f88" path="/var/lib/kubelet/pods/c3de5bf3-0919-4e77-ab84-35ec6f218f88/volumes" Oct 30 07:34:16.341915 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.341874 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:34:16.396940 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.396901 4951 generic.go:356] "Generic (PLEG): container finished" podID="2ea0e265-4a3b-46ff-98d1-129054c698df" containerID="a75ce083500fd2e5fa40312feaef96567507cceab1f54a5781c4db842454b370" exitCode=0 Oct 30 07:34:16.397142 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.396955 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" event={"ID":"2ea0e265-4a3b-46ff-98d1-129054c698df","Type":"ContainerDied","Data":"a75ce083500fd2e5fa40312feaef96567507cceab1f54a5781c4db842454b370"} Oct 30 07:34:16.397142 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.396968 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" Oct 30 07:34:16.397142 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.397053 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54" event={"ID":"2ea0e265-4a3b-46ff-98d1-129054c698df","Type":"ContainerDied","Data":"d10aa56fc6f0de22ebc4d8ff8dfcc854b2e5305fa071123a6b3e1cb53df612f4"} Oct 30 07:34:16.397142 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.397085 4951 scope.go:122] "RemoveContainer" containerID="a75ce083500fd2e5fa40312feaef96567507cceab1f54a5781c4db842454b370" Oct 30 07:34:16.407862 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.407840 4951 scope.go:122] "RemoveContainer" containerID="a75ce083500fd2e5fa40312feaef96567507cceab1f54a5781c4db842454b370" Oct 30 07:34:16.408125 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:34:16.408106 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a75ce083500fd2e5fa40312feaef96567507cceab1f54a5781c4db842454b370\": container with ID starting with a75ce083500fd2e5fa40312feaef96567507cceab1f54a5781c4db842454b370 not found: ID does not exist" containerID="a75ce083500fd2e5fa40312feaef96567507cceab1f54a5781c4db842454b370" Oct 30 07:34:16.408192 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.408134 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a75ce083500fd2e5fa40312feaef96567507cceab1f54a5781c4db842454b370"} err="failed to get container status \"a75ce083500fd2e5fa40312feaef96567507cceab1f54a5781c4db842454b370\": rpc error: code = NotFound desc = could not find container \"a75ce083500fd2e5fa40312feaef96567507cceab1f54a5781c4db842454b370\": container with ID starting with a75ce083500fd2e5fa40312feaef96567507cceab1f54a5781c4db842454b370 not found: ID does not exist" Oct 30 07:34:16.418729 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.418586 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-istio-envoy\") pod \"2ea0e265-4a3b-46ff-98d1-129054c698df\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " Oct 30 07:34:16.418729 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.418661 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ea0e265-4a3b-46ff-98d1-129054c698df-istio-token\" (UniqueName: \"kubernetes.io/projected/2ea0e265-4a3b-46ff-98d1-129054c698df-istio-token\") pod \"2ea0e265-4a3b-46ff-98d1-129054c698df\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " Oct 30 07:34:16.418729 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.418723 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-istio-data\") pod \"2ea0e265-4a3b-46ff-98d1-129054c698df\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " Oct 30 07:34:16.418989 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.418842 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ea0e265-4a3b-46ff-98d1-129054c698df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ea0e265-4a3b-46ff-98d1-129054c698df-istio-podinfo\") pod \"2ea0e265-4a3b-46ff-98d1-129054c698df\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " Oct 30 07:34:16.418989 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.418883 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-workload-certs\") pod \"2ea0e265-4a3b-46ff-98d1-129054c698df\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " Oct 30 07:34:16.418989 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.418920 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-credential-socket\") pod \"2ea0e265-4a3b-46ff-98d1-129054c698df\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " Oct 30 07:34:16.418989 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.418986 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ea0e265-4a3b-46ff-98d1-129054c698df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ea0e265-4a3b-46ff-98d1-129054c698df-istiod-ca-cert\") pod \"2ea0e265-4a3b-46ff-98d1-129054c698df\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " Oct 30 07:34:16.419235 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.419018 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ea0e265-4a3b-46ff-98d1-129054c698df-kube-api-access-w22j7\" (UniqueName: \"kubernetes.io/projected/2ea0e265-4a3b-46ff-98d1-129054c698df-kube-api-access-w22j7\") pod \"2ea0e265-4a3b-46ff-98d1-129054c698df\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " Oct 30 07:34:16.419235 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.419125 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-workload-socket\") pod \"2ea0e265-4a3b-46ff-98d1-129054c698df\" (UID: \"2ea0e265-4a3b-46ff-98d1-129054c698df\") " Oct 30 07:34:16.419766 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.419738 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-istio-data" pod "2ea0e265-4a3b-46ff-98d1-129054c698df" (UID: "2ea0e265-4a3b-46ff-98d1-129054c698df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:34:16.420042 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.420020 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-credential-socket" pod "2ea0e265-4a3b-46ff-98d1-129054c698df" (UID: "2ea0e265-4a3b-46ff-98d1-129054c698df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:34:16.420386 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.420359 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ea0e265-4a3b-46ff-98d1-129054c698df-istiod-ca-cert" pod "2ea0e265-4a3b-46ff-98d1-129054c698df" (UID: "2ea0e265-4a3b-46ff-98d1-129054c698df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:34:16.420539 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.420518 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-workload-certs" pod "2ea0e265-4a3b-46ff-98d1-129054c698df" (UID: "2ea0e265-4a3b-46ff-98d1-129054c698df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:34:16.420769 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.420730 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-workload-socket" pod "2ea0e265-4a3b-46ff-98d1-129054c698df" (UID: "2ea0e265-4a3b-46ff-98d1-129054c698df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:34:16.422211 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.422184 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ea0e265-4a3b-46ff-98d1-129054c698df-istio-podinfo" pod "2ea0e265-4a3b-46ff-98d1-129054c698df" (UID: "2ea0e265-4a3b-46ff-98d1-129054c698df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:34:16.422301 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.422257 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea0e265-4a3b-46ff-98d1-129054c698df-kube-api-access-w22j7" pod "2ea0e265-4a3b-46ff-98d1-129054c698df" (UID: "2ea0e265-4a3b-46ff-98d1-129054c698df"). InnerVolumeSpecName "kube-api-access-w22j7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:34:16.422442 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.422419 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-istio-envoy" pod "2ea0e265-4a3b-46ff-98d1-129054c698df" (UID: "2ea0e265-4a3b-46ff-98d1-129054c698df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:34:16.422565 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.422546 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea0e265-4a3b-46ff-98d1-129054c698df-istio-token" pod "2ea0e265-4a3b-46ff-98d1-129054c698df" (UID: "2ea0e265-4a3b-46ff-98d1-129054c698df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:34:16.521187 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.521150 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ea0e265-4a3b-46ff-98d1-129054c698df-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:34:16.521187 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.521183 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:34:16.521425 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.521196 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:34:16.521425 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.521210 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ea0e265-4a3b-46ff-98d1-129054c698df-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:34:16.521425 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.521222 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w22j7\" (UniqueName: \"kubernetes.io/projected/2ea0e265-4a3b-46ff-98d1-129054c698df-kube-api-access-w22j7\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:34:16.521425 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.521235 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:34:16.521425 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.521246 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:34:16.521425 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.521257 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ea0e265-4a3b-46ff-98d1-129054c698df-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:34:16.521425 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.521268 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ea0e265-4a3b-46ff-98d1-129054c698df-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:34:16.750351 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.750311 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54"] Oct 30 07:34:16.756240 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:16.756205 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-bbl54"] Oct 30 07:34:17.774815 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:17.774778 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ea0e265-4a3b-46ff-98d1-129054c698df" path="/var/lib/kubelet/pods/2ea0e265-4a3b-46ff-98d1-129054c698df/volumes" Oct 30 07:34:17.858872 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:17.858838 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn"] Oct 30 07:34:17.859191 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:17.859159 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" podUID="a4fcbdd5-1641-44fe-9a29-0be0aceb6129" containerName="istio-proxy" containerID="cri-o://30a031e84d7757a8d50323e3faeef05357e5472d8358f52efb4aa6dd03d3860d" gracePeriod=30 Oct 30 07:34:18.459742 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:18.459707 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7593-daf496"] Oct 30 07:34:18.460687 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:18.460667 4951 cpu_manager.go:414] "containerMap: removing container" podUID="60fe230d-06d4-4c8a-ba90-71b87281216f" containerName="payload-pre-processing" Oct 30 07:34:18.460752 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:18.460690 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60fe230d-06d4-4c8a-ba90-71b87281216f" containerName="payload-pre-processing" Oct 30 07:34:18.460752 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:18.460730 4951 cpu_manager.go:414] "containerMap: removing container" podUID="da92d4f9-c801-4fc9-9eaf-589305c6851d" containerName="istio-proxy" Oct 30 07:34:18.460752 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:18.460738 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da92d4f9-c801-4fc9-9eaf-589305c6851d" containerName="istio-proxy" Oct 30 07:34:18.460917 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:18.460764 4951 cpu_manager.go:414] "containerMap: removing container" podUID="088ac46a-3235-420e-9680-74cb738f229c" containerName="istio-proxy" Oct 30 07:34:18.460917 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:18.460772 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="088ac46a-3235-420e-9680-74cb738f229c" containerName="istio-proxy" Oct 30 07:34:18.460917 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:18.460785 4951 cpu_manager.go:414] "containerMap: removing container" podUID="2ea0e265-4a3b-46ff-98d1-129054c698df" containerName="istio-proxy" Oct 30 07:34:18.460917 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:18.460793 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea0e265-4a3b-46ff-98d1-129054c698df" containerName="istio-proxy" Oct 30 07:34:18.460917 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:18.460809 4951 cpu_manager.go:414] "containerMap: removing container" podUID="246b0113-0e73-46b7-88f6-3fac16df57b5" containerName="istio-proxy" Oct 30 07:34:18.460917 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:18.460816 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="246b0113-0e73-46b7-88f6-3fac16df57b5" containerName="istio-proxy" Oct 30 07:34:18.460917 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:18.460837 4951 cpu_manager.go:414] "containerMap: removing container" podUID="ea726207-9a8b-4494-bf06-c339b4114002" containerName="istio-proxy" Oct 30 07:34:18.460917 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:18.460846 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea726207-9a8b-4494-bf06-c339b4114002" containerName="istio-proxy" Oct 30 07:34:18.460917 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:18.460864 4951 cpu_manager.go:414] "containerMap: removing container" podUID="c3de5bf3-0919-4e77-ab84-35ec6f218f88" containerName="payload-processing" Oct 30 07:34:18.460917 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:18.460871 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3de5bf3-0919-4e77-ab84-35ec6f218f88" containerName="payload-processing" Oct 30 07:34:18.461230 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:18.461001 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea726207-9a8b-4494-bf06-c339b4114002" containerName="istio-proxy" Oct 30 07:34:18.461230 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:18.461013 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3de5bf3-0919-4e77-ab84-35ec6f218f88" containerName="payload-processing" Oct 30 07:34:18.461230 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:18.461027 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="60fe230d-06d4-4c8a-ba90-71b87281216f" containerName="payload-pre-processing" Oct 30 07:34:18.461230 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:18.461046 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="da92d4f9-c801-4fc9-9eaf-589305c6851d" containerName="istio-proxy" Oct 30 07:34:18.461230 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:18.461055 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="088ac46a-3235-420e-9680-74cb738f229c" containerName="istio-proxy" Oct 30 07:34:18.461230 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:18.461065 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ea0e265-4a3b-46ff-98d1-129054c698df" containerName="istio-proxy" Oct 30 07:34:18.461230 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:18.461076 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="246b0113-0e73-46b7-88f6-3fac16df57b5" containerName="istio-proxy" Oct 30 07:34:18.466031 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:18.466010 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-daf496" Oct 30 07:34:18.480884 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:18.480849 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7593-daf496"] Oct 30 07:34:18.482612 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:18.482556 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5z8jp\"" Oct 30 07:34:18.567371 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:18.567331 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7qcs\" (UniqueName: \"kubernetes.io/projected/c183c5e4-e101-48a0-b215-c536e9c384d7-kube-api-access-r7qcs\") pod \"test-curl-7593-daf496\" (UID: \"c183c5e4-e101-48a0-b215-c536e9c384d7\") " pod="opendatahub/test-curl-7593-daf496" Oct 30 07:34:18.669130 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:18.669092 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7qcs\" (UniqueName: \"kubernetes.io/projected/c183c5e4-e101-48a0-b215-c536e9c384d7-kube-api-access-r7qcs\") pod \"test-curl-7593-daf496\" (UID: \"c183c5e4-e101-48a0-b215-c536e9c384d7\") " pod="opendatahub/test-curl-7593-daf496" Oct 30 07:34:18.694416 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:18.694382 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7qcs\" (UniqueName: \"kubernetes.io/projected/c183c5e4-e101-48a0-b215-c536e9c384d7-kube-api-access-r7qcs\") pod \"test-curl-7593-daf496\" (UID: \"c183c5e4-e101-48a0-b215-c536e9c384d7\") " pod="opendatahub/test-curl-7593-daf496" Oct 30 07:34:18.779016 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:18.778975 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-daf496" Oct 30 07:34:19.115254 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:19.113390 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7593-daf496"] Oct 30 07:34:19.436247 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:19.436133 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7593-daf496" event={"ID":"c183c5e4-e101-48a0-b215-c536e9c384d7","Type":"ContainerStarted","Data":"4316126ee1d0cc54d262fc05fa80f4b23f7c7246b58cf63d62e0463fa0dc9a52"} Oct 30 07:34:19.436247 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:19.436191 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7593-daf496" event={"ID":"c183c5e4-e101-48a0-b215-c536e9c384d7","Type":"ContainerStarted","Data":"7793bcd43905583dddc2f40adda0ab3607987658e1ff8da24eaca2bcb7e43c2b"} Oct 30 07:34:19.478236 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:19.478188 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7593-daf496" podStartSLOduration=1.478174165 podStartE2EDuration="1.478174165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:34:19.476709898 +0000 UTC m=+2265.819757863" watchObservedRunningTime="2026-10-30 07:34:19.478174165 +0000 UTC m=+2265.821222129" Oct 30 07:34:20.393778 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:20.393745 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7593-daf496"] Oct 30 07:34:20.401149 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:20.401117 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7593-daf496"] Oct 30 07:34:21.452626 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.452527 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7593-daf496" podUID="c183c5e4-e101-48a0-b215-c536e9c384d7" containerName="test-curl-7593-daf496" containerID="cri-o://4316126ee1d0cc54d262fc05fa80f4b23f7c7246b58cf63d62e0463fa0dc9a52" gracePeriod=2 Oct 30 07:34:21.467769 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:34:21.467726 4951 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7593-daf496\" is forbidden: User \"system:node:ip-10-0-132-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-173.ec2.internal' and this object" podUID="c183c5e4-e101-48a0-b215-c536e9c384d7" pod="opendatahub/test-curl-7593-daf496" Oct 30 07:34:21.561871 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.561840 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x"] Oct 30 07:34:21.562541 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.562525 4951 cpu_manager.go:414] "containerMap: removing container" podUID="c183c5e4-e101-48a0-b215-c536e9c384d7" containerName="test-curl-7593-daf496" Oct 30 07:34:21.562632 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.562543 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c183c5e4-e101-48a0-b215-c536e9c384d7" containerName="test-curl-7593-daf496" Oct 30 07:34:21.562717 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.562706 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="c183c5e4-e101-48a0-b215-c536e9c384d7" containerName="test-curl-7593-daf496" Oct 30 07:34:21.566276 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.566257 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:34:21.582372 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:34:21.582332 4951 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7593-daf496\" is forbidden: User \"system:node:ip-10-0-132-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-173.ec2.internal' and this object" podUID="c183c5e4-e101-48a0-b215-c536e9c384d7" pod="opendatahub/test-curl-7593-daf496" Oct 30 07:34:21.583802 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.583782 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-54f2b2-openshift-default-dockercfg-p88bf\"" Oct 30 07:34:21.598043 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.598008 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x"] Oct 30 07:34:21.778593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.770442 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istio-token\") pod \"e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:34:21.778593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.770494 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvnlf\" (UniqueName: \"kubernetes.io/projected/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-kube-api-access-cvnlf\") pod \"e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:34:21.778593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.770602 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istio-envoy\") pod \"e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:34:21.778593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.770639 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istiod-ca-cert\") pod \"e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:34:21.778593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.771012 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istio-podinfo\") pod \"e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:34:21.778593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.771062 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-credential-socket\") pod \"e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:34:21.778593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.771100 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istio-data\") pod \"e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:34:21.778593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.771128 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-workload-socket\") pod \"e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:34:21.778593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.771191 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-workload-certs\") pod \"e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:34:21.878593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.877113 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-workload-certs\") pod \"e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:34:21.878593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.877262 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istio-token\") pod \"e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:34:21.878593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.877290 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvnlf\" (UniqueName: \"kubernetes.io/projected/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-kube-api-access-cvnlf\") pod \"e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:34:21.878593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.877389 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istio-envoy\") pod \"e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:34:21.878593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.877422 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istiod-ca-cert\") pod \"e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:34:21.878593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.877508 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istio-podinfo\") pod \"e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:34:21.878593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.877556 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-credential-socket\") pod \"e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:34:21.878593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.877618 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istio-data\") pod \"e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:34:21.878593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.877649 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-workload-socket\") pod \"e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:34:21.878593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.878219 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-workload-socket\") pod \"e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:34:21.878593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.878519 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-workload-certs\") pod \"e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:34:21.881002 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.880966 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istiod-ca-cert\") pod \"e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:34:21.914594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.881435 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-credential-socket\") pod \"e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:34:21.914594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.881985 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istio-data\") pod \"e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:34:21.914594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.899116 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istio-podinfo\") pod \"e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:34:21.919170 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.918641 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istio-envoy\") pod \"e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:34:21.921959 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.921904 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istio-token\") pod \"e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:34:21.922098 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:21.922073 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvnlf\" (UniqueName: \"kubernetes.io/projected/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-kube-api-access-cvnlf\") pod \"e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:34:22.183287 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:22.183206 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:34:22.533707 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:34:22.533667 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8562e54e_5cb6_4805_ae61_afc7b1d5c0c6.slice/crio-bf3411f6363fa8db46d40e9705f509656da806cee14213bdf147da35220d46f9 WatchSource:0}: Error finding container bf3411f6363fa8db46d40e9705f509656da806cee14213bdf147da35220d46f9: Status 404 returned error can't find the container with id bf3411f6363fa8db46d40e9705f509656da806cee14213bdf147da35220d46f9 Oct 30 07:34:22.535869 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:22.535835 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x"] Oct 30 07:34:22.537593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:22.537418 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:34:22.537593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:22.537519 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:34:22.537593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:22.537562 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:34:23.027983 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:34:23.027951 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:34:23.155797 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.155766 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:34:23.198023 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.197990 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istio-envoy\") pod \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " Oct 30 07:34:23.198215 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.198076 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istio-data\") pod \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " Oct 30 07:34:23.198215 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.198107 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istiod-ca-cert\") pod \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " Oct 30 07:34:23.198215 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.198159 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-workload-socket\") pod \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " Oct 30 07:34:23.198215 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.198207 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istio-token\" (UniqueName: \"kubernetes.io/projected/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istio-token\") pod \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " Oct 30 07:34:23.198432 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.198265 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istio-podinfo\") pod \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " Oct 30 07:34:23.198432 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.198340 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-workload-certs\") pod \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " Oct 30 07:34:23.198432 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.198355 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istio-data" pod "a4fcbdd5-1641-44fe-9a29-0be0aceb6129" (UID: "a4fcbdd5-1641-44fe-9a29-0be0aceb6129"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:34:23.198630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.198431 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-kube-api-access-vfkjs\" (UniqueName: \"kubernetes.io/projected/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-kube-api-access-vfkjs\") pod \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " Oct 30 07:34:23.198630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.198518 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-credential-socket\") pod \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\" (UID: \"a4fcbdd5-1641-44fe-9a29-0be0aceb6129\") " Oct 30 07:34:23.198630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.198538 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istiod-ca-cert" pod "a4fcbdd5-1641-44fe-9a29-0be0aceb6129" (UID: "a4fcbdd5-1641-44fe-9a29-0be0aceb6129"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:34:23.198813 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.198656 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-workload-certs" pod "a4fcbdd5-1641-44fe-9a29-0be0aceb6129" (UID: "a4fcbdd5-1641-44fe-9a29-0be0aceb6129"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:34:23.199023 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.198999 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-credential-socket" pod "a4fcbdd5-1641-44fe-9a29-0be0aceb6129" (UID: "a4fcbdd5-1641-44fe-9a29-0be0aceb6129"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:34:23.199095 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.199020 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-workload-socket" pod "a4fcbdd5-1641-44fe-9a29-0be0aceb6129" (UID: "a4fcbdd5-1641-44fe-9a29-0be0aceb6129"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:34:23.199226 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.199182 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:34:23.199226 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.199204 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:34:23.199226 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.199219 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:34:23.199401 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.199231 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:34:23.199401 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.199245 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:34:23.200902 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.200839 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istio-envoy" pod "a4fcbdd5-1641-44fe-9a29-0be0aceb6129" (UID: "a4fcbdd5-1641-44fe-9a29-0be0aceb6129"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:34:23.201182 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.201156 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istio-token" pod "a4fcbdd5-1641-44fe-9a29-0be0aceb6129" (UID: "a4fcbdd5-1641-44fe-9a29-0be0aceb6129"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:34:23.202064 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.202030 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istio-podinfo" pod "a4fcbdd5-1641-44fe-9a29-0be0aceb6129" (UID: "a4fcbdd5-1641-44fe-9a29-0be0aceb6129"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:34:23.202179 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.202066 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-kube-api-access-vfkjs" pod "a4fcbdd5-1641-44fe-9a29-0be0aceb6129" (UID: "a4fcbdd5-1641-44fe-9a29-0be0aceb6129"). InnerVolumeSpecName "kube-api-access-vfkjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:34:23.301110 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.301073 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:34:23.301110 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.301108 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:34:23.301329 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.301122 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:34:23.301329 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.301137 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfkjs\" (UniqueName: \"kubernetes.io/projected/a4fcbdd5-1641-44fe-9a29-0be0aceb6129-kube-api-access-vfkjs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:34:23.468490 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.468464 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" event={"ID":"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6","Type":"ContainerStarted","Data":"298585ce83863ba43af616efbe835894bca8b711581bdc06ca7bff1d3ab975b7"} Oct 30 07:34:23.468631 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.468524 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" event={"ID":"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6","Type":"ContainerStarted","Data":"bf3411f6363fa8db46d40e9705f509656da806cee14213bdf147da35220d46f9"} Oct 30 07:34:23.470267 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.470245 4951 generic.go:356] "Generic (PLEG): container finished" podID="a4fcbdd5-1641-44fe-9a29-0be0aceb6129" containerID="30a031e84d7757a8d50323e3faeef05357e5472d8358f52efb4aa6dd03d3860d" exitCode=0 Oct 30 07:34:23.470393 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.470290 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" Oct 30 07:34:23.470393 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.470316 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" event={"ID":"a4fcbdd5-1641-44fe-9a29-0be0aceb6129","Type":"ContainerDied","Data":"30a031e84d7757a8d50323e3faeef05357e5472d8358f52efb4aa6dd03d3860d"} Oct 30 07:34:23.470393 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.470358 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn" event={"ID":"a4fcbdd5-1641-44fe-9a29-0be0aceb6129","Type":"ContainerDied","Data":"8609f0d16befb285494ebbc445e9a7e17b70228fdc88b210b27bc2cf9f6d421b"} Oct 30 07:34:23.470393 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.470379 4951 scope.go:122] "RemoveContainer" containerID="30a031e84d7757a8d50323e3faeef05357e5472d8358f52efb4aa6dd03d3860d" Oct 30 07:34:23.559605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.558790 4951 scope.go:122] "RemoveContainer" containerID="5fb867544af00246958ab89e6516b37da0c80c9bc9badd3336b2b9f58173f21a" Oct 30 07:34:23.607189 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.607123 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" podStartSLOduration=2.607103187 podStartE2EDuration="2.607103187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:34:23.534439173 +0000 UTC m=+2269.877487142" watchObservedRunningTime="2026-10-30 07:34:23.607103187 +0000 UTC m=+2269.950151156" Oct 30 07:34:23.608769 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.608742 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn"] Oct 30 07:34:23.620625 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.614633 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-lxqqn"] Oct 30 07:34:23.670761 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.660312 4951 scope.go:122] "RemoveContainer" containerID="30a031e84d7757a8d50323e3faeef05357e5472d8358f52efb4aa6dd03d3860d" Oct 30 07:34:23.696597 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:34:23.690097 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30a031e84d7757a8d50323e3faeef05357e5472d8358f52efb4aa6dd03d3860d\": container with ID starting with 30a031e84d7757a8d50323e3faeef05357e5472d8358f52efb4aa6dd03d3860d not found: ID does not exist" containerID="30a031e84d7757a8d50323e3faeef05357e5472d8358f52efb4aa6dd03d3860d" Oct 30 07:34:23.696597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.690153 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30a031e84d7757a8d50323e3faeef05357e5472d8358f52efb4aa6dd03d3860d"} err="failed to get container status \"30a031e84d7757a8d50323e3faeef05357e5472d8358f52efb4aa6dd03d3860d\": rpc error: code = NotFound desc = could not find container \"30a031e84d7757a8d50323e3faeef05357e5472d8358f52efb4aa6dd03d3860d\": container with ID starting with 30a031e84d7757a8d50323e3faeef05357e5472d8358f52efb4aa6dd03d3860d not found: ID does not exist" Oct 30 07:34:23.696597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.690185 4951 scope.go:122] "RemoveContainer" containerID="5fb867544af00246958ab89e6516b37da0c80c9bc9badd3336b2b9f58173f21a" Oct 30 07:34:23.696597 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:34:23.693933 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fb867544af00246958ab89e6516b37da0c80c9bc9badd3336b2b9f58173f21a\": container with ID starting with 5fb867544af00246958ab89e6516b37da0c80c9bc9badd3336b2b9f58173f21a not found: ID does not exist" containerID="5fb867544af00246958ab89e6516b37da0c80c9bc9badd3336b2b9f58173f21a" Oct 30 07:34:23.696597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.693975 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb867544af00246958ab89e6516b37da0c80c9bc9badd3336b2b9f58173f21a"} err="failed to get container status \"5fb867544af00246958ab89e6516b37da0c80c9bc9badd3336b2b9f58173f21a\": rpc error: code = NotFound desc = could not find container \"5fb867544af00246958ab89e6516b37da0c80c9bc9badd3336b2b9f58173f21a\": container with ID starting with 5fb867544af00246958ab89e6516b37da0c80c9bc9badd3336b2b9f58173f21a not found: ID does not exist" Oct 30 07:34:23.779519 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.779476 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4fcbdd5-1641-44fe-9a29-0be0aceb6129" path="/var/lib/kubelet/pods/a4fcbdd5-1641-44fe-9a29-0be0aceb6129/volumes" Oct 30 07:34:23.869794 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.869767 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-daf496" Oct 30 07:34:23.885193 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:34:23.885153 4951 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7593-daf496\" is forbidden: User \"system:node:ip-10-0-132-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-173.ec2.internal' and this object" podUID="c183c5e4-e101-48a0-b215-c536e9c384d7" pod="opendatahub/test-curl-7593-daf496" Oct 30 07:34:23.963007 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.962981 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c183c5e4-e101-48a0-b215-c536e9c384d7-kube-api-access-r7qcs\" (UniqueName: \"kubernetes.io/projected/c183c5e4-e101-48a0-b215-c536e9c384d7-kube-api-access-r7qcs\") pod \"c183c5e4-e101-48a0-b215-c536e9c384d7\" (UID: \"c183c5e4-e101-48a0-b215-c536e9c384d7\") " Oct 30 07:34:23.965274 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:23.965246 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c183c5e4-e101-48a0-b215-c536e9c384d7-kube-api-access-r7qcs" pod "c183c5e4-e101-48a0-b215-c536e9c384d7" (UID: "c183c5e4-e101-48a0-b215-c536e9c384d7"). InnerVolumeSpecName "kube-api-access-r7qcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:34:24.068645 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:24.065911 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7qcs\" (UniqueName: \"kubernetes.io/projected/c183c5e4-e101-48a0-b215-c536e9c384d7-kube-api-access-r7qcs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:34:24.190596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:24.189947 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:34:24.211593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:24.209958 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:34:24.227842 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:34:24.227796 4951 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7593-daf496\" is forbidden: User \"system:node:ip-10-0-132-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-173.ec2.internal' and this object" podUID="c183c5e4-e101-48a0-b215-c536e9c384d7" pod="opendatahub/test-curl-7593-daf496" Oct 30 07:34:24.482442 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:24.482349 4951 generic.go:356] "Generic (PLEG): container finished" podID="c183c5e4-e101-48a0-b215-c536e9c384d7" containerID="4316126ee1d0cc54d262fc05fa80f4b23f7c7246b58cf63d62e0463fa0dc9a52" exitCode=137 Oct 30 07:34:24.482442 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:24.482415 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-daf496" Oct 30 07:34:24.482744 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:24.482457 4951 scope.go:122] "RemoveContainer" containerID="4316126ee1d0cc54d262fc05fa80f4b23f7c7246b58cf63d62e0463fa0dc9a52" Oct 30 07:34:24.483517 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:24.483443 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:34:24.486943 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:24.486920 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:34:24.498210 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:34:24.497979 4951 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7593-daf496\" is forbidden: User \"system:node:ip-10-0-132-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-173.ec2.internal' and this object" podUID="c183c5e4-e101-48a0-b215-c536e9c384d7" pod="opendatahub/test-curl-7593-daf496" Oct 30 07:34:24.499252 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:24.499225 4951 scope.go:122] "RemoveContainer" containerID="4316126ee1d0cc54d262fc05fa80f4b23f7c7246b58cf63d62e0463fa0dc9a52" Oct 30 07:34:24.499641 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:34:24.499612 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4316126ee1d0cc54d262fc05fa80f4b23f7c7246b58cf63d62e0463fa0dc9a52\": container with ID starting with 4316126ee1d0cc54d262fc05fa80f4b23f7c7246b58cf63d62e0463fa0dc9a52 not found: ID does not exist" containerID="4316126ee1d0cc54d262fc05fa80f4b23f7c7246b58cf63d62e0463fa0dc9a52" Oct 30 07:34:24.499758 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:24.499652 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4316126ee1d0cc54d262fc05fa80f4b23f7c7246b58cf63d62e0463fa0dc9a52"} err="failed to get container status \"4316126ee1d0cc54d262fc05fa80f4b23f7c7246b58cf63d62e0463fa0dc9a52\": rpc error: code = NotFound desc = could not find container \"4316126ee1d0cc54d262fc05fa80f4b23f7c7246b58cf63d62e0463fa0dc9a52\": container with ID starting with 4316126ee1d0cc54d262fc05fa80f4b23f7c7246b58cf63d62e0463fa0dc9a52 not found: ID does not exist" Oct 30 07:34:24.596493 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:34:24.595871 4951 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7593-daf496\" is forbidden: User \"system:node:ip-10-0-132-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-173.ec2.internal' and this object" podUID="c183c5e4-e101-48a0-b215-c536e9c384d7" pod="opendatahub/test-curl-7593-daf496" Oct 30 07:34:24.611852 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:34:24.611805 4951 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7593-daf496\" is forbidden: User \"system:node:ip-10-0-132-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-173.ec2.internal' and this object" podUID="c183c5e4-e101-48a0-b215-c536e9c384d7" pod="opendatahub/test-curl-7593-daf496" Oct 30 07:34:25.772657 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:25.772605 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c183c5e4-e101-48a0-b215-c536e9c384d7" path="/var/lib/kubelet/pods/c183c5e4-e101-48a0-b215-c536e9c384d7/volumes" Oct 30 07:34:27.998594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:27.998536 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-54f2b2-59f695d955-8fjrt"] Oct 30 07:34:27.999743 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:27.999714 4951 cpu_manager.go:414] "containerMap: removing container" podUID="a4fcbdd5-1641-44fe-9a29-0be0aceb6129" containerName="istio-proxy" Oct 30 07:34:27.999743 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:27.999738 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4fcbdd5-1641-44fe-9a29-0be0aceb6129" containerName="istio-proxy" Oct 30 07:34:27.999949 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:27.999910 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4fcbdd5-1641-44fe-9a29-0be0aceb6129" containerName="istio-proxy" Oct 30 07:34:27.999949 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:27.999931 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4fcbdd5-1641-44fe-9a29-0be0aceb6129" containerName="istio-proxy" Oct 30 07:34:28.005529 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:28.005484 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-54f2b2-59f695d955-8fjrt" Oct 30 07:34:28.040151 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:28.040103 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-54f2b2-dockercfg-v2gqn\"" Oct 30 07:34:28.040522 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:28.040491 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-54f2b2\"" Oct 30 07:34:28.043972 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:28.043945 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-54f2b2-59f695d955-8fjrt"] Oct 30 07:34:28.081953 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:28.081916 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-54f2b2-766bc666b6-ljwdv"] Oct 30 07:34:28.082695 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:28.082664 4951 cpu_manager.go:414] "containerMap: removing container" podUID="a4fcbdd5-1641-44fe-9a29-0be0aceb6129" containerName="istio-proxy" Oct 30 07:34:28.082695 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:28.082689 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4fcbdd5-1641-44fe-9a29-0be0aceb6129" containerName="istio-proxy" Oct 30 07:34:28.280282 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:28.280218 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64glc\" (UniqueName: \"kubernetes.io/projected/4dd49e47-404e-4be7-8e30-26f4ceb2bc1b-kube-api-access-64glc\") pod \"payload-pre-processing-e2e-shared-a-w0-54f2b2-59f695d955-8fjrt\" (UID: \"4dd49e47-404e-4be7-8e30-26f4ceb2bc1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-54f2b2-59f695d955-8fjrt" Oct 30 07:34:28.280627 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:28.280606 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dd49e47-404e-4be7-8e30-26f4ceb2bc1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-54f2b2-59f695d955-8fjrt\" (UID: \"4dd49e47-404e-4be7-8e30-26f4ceb2bc1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-54f2b2-59f695d955-8fjrt" Oct 30 07:34:28.288030 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:28.284523 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-54f2b2-766bc666b6-ljwdv" Oct 30 07:34:28.311595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:28.290676 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-54f2b2-766bc666b6-ljwdv"] Oct 30 07:34:28.387645 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:28.387599 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64glc\" (UniqueName: \"kubernetes.io/projected/4dd49e47-404e-4be7-8e30-26f4ceb2bc1b-kube-api-access-64glc\") pod \"payload-pre-processing-e2e-shared-a-w0-54f2b2-59f695d955-8fjrt\" (UID: \"4dd49e47-404e-4be7-8e30-26f4ceb2bc1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-54f2b2-59f695d955-8fjrt" Oct 30 07:34:28.387645 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:28.387658 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dd49e47-404e-4be7-8e30-26f4ceb2bc1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-54f2b2-59f695d955-8fjrt\" (UID: \"4dd49e47-404e-4be7-8e30-26f4ceb2bc1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-54f2b2-59f695d955-8fjrt" Oct 30 07:34:28.409688 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:28.399934 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dd49e47-404e-4be7-8e30-26f4ceb2bc1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-54f2b2-59f695d955-8fjrt\" (UID: \"4dd49e47-404e-4be7-8e30-26f4ceb2bc1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-54f2b2-59f695d955-8fjrt" Oct 30 07:34:28.427645 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:28.426647 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64glc\" (UniqueName: \"kubernetes.io/projected/4dd49e47-404e-4be7-8e30-26f4ceb2bc1b-kube-api-access-64glc\") pod \"payload-pre-processing-e2e-shared-a-w0-54f2b2-59f695d955-8fjrt\" (UID: \"4dd49e47-404e-4be7-8e30-26f4ceb2bc1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-54f2b2-59f695d955-8fjrt" Oct 30 07:34:28.489027 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:28.488987 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhzvh\" (UniqueName: \"kubernetes.io/projected/0f9d9e31-a0d8-4c02-93d9-5dbe88167315-kube-api-access-bhzvh\") pod \"payload-processing-e2e-shared-a-w0-54f2b2-766bc666b6-ljwdv\" (UID: \"0f9d9e31-a0d8-4c02-93d9-5dbe88167315\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-54f2b2-766bc666b6-ljwdv" Oct 30 07:34:28.489239 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:28.489167 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f9d9e31-a0d8-4c02-93d9-5dbe88167315-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-54f2b2-766bc666b6-ljwdv\" (UID: \"0f9d9e31-a0d8-4c02-93d9-5dbe88167315\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-54f2b2-766bc666b6-ljwdv" Oct 30 07:34:28.590430 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:28.590344 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhzvh\" (UniqueName: \"kubernetes.io/projected/0f9d9e31-a0d8-4c02-93d9-5dbe88167315-kube-api-access-bhzvh\") pod \"payload-processing-e2e-shared-a-w0-54f2b2-766bc666b6-ljwdv\" (UID: \"0f9d9e31-a0d8-4c02-93d9-5dbe88167315\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-54f2b2-766bc666b6-ljwdv" Oct 30 07:34:28.590627 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:28.590460 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f9d9e31-a0d8-4c02-93d9-5dbe88167315-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-54f2b2-766bc666b6-ljwdv\" (UID: \"0f9d9e31-a0d8-4c02-93d9-5dbe88167315\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-54f2b2-766bc666b6-ljwdv" Oct 30 07:34:28.591193 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:28.591173 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f9d9e31-a0d8-4c02-93d9-5dbe88167315-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-54f2b2-766bc666b6-ljwdv\" (UID: \"0f9d9e31-a0d8-4c02-93d9-5dbe88167315\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-54f2b2-766bc666b6-ljwdv" Oct 30 07:34:28.621699 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:28.621642 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-54f2b2-59f695d955-8fjrt" Oct 30 07:34:28.628291 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:28.628260 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhzvh\" (UniqueName: \"kubernetes.io/projected/0f9d9e31-a0d8-4c02-93d9-5dbe88167315-kube-api-access-bhzvh\") pod \"payload-processing-e2e-shared-a-w0-54f2b2-766bc666b6-ljwdv\" (UID: \"0f9d9e31-a0d8-4c02-93d9-5dbe88167315\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-54f2b2-766bc666b6-ljwdv" Oct 30 07:34:28.640692 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:28.640662 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-54f2b2-766bc666b6-ljwdv" Oct 30 07:34:28.936754 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:34:28.936720 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dd49e47_404e_4be7_8e30_26f4ceb2bc1b.slice/crio-1f17dce412acaf3c46e2e106071d8eb93f83bea78b9f9cbfa894c4de4d7271f5 WatchSource:0}: Error finding container 1f17dce412acaf3c46e2e106071d8eb93f83bea78b9f9cbfa894c4de4d7271f5: Status 404 returned error can't find the container with id 1f17dce412acaf3c46e2e106071d8eb93f83bea78b9f9cbfa894c4de4d7271f5 Oct 30 07:34:28.939373 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:28.939347 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-54f2b2-59f695d955-8fjrt"] Oct 30 07:34:28.965139 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:28.965110 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-54f2b2-766bc666b6-ljwdv"] Oct 30 07:34:28.968047 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:34:28.968011 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f9d9e31_a0d8_4c02_93d9_5dbe88167315.slice/crio-3b734cf016531a00e10ec9418dafae0983eb8caef8686d7d0c2e0d8eae94981a WatchSource:0}: Error finding container 3b734cf016531a00e10ec9418dafae0983eb8caef8686d7d0c2e0d8eae94981a: Status 404 returned error can't find the container with id 3b734cf016531a00e10ec9418dafae0983eb8caef8686d7d0c2e0d8eae94981a Oct 30 07:34:29.523716 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:29.523677 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-54f2b2-766bc666b6-ljwdv" event={"ID":"0f9d9e31-a0d8-4c02-93d9-5dbe88167315","Type":"ContainerStarted","Data":"c63dc0ea79641a69e7fe894feae5a78c7a55b0ed7733ef55a1d820ebc59bc487"} Oct 30 07:34:29.524162 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:29.523740 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-54f2b2-766bc666b6-ljwdv" event={"ID":"0f9d9e31-a0d8-4c02-93d9-5dbe88167315","Type":"ContainerStarted","Data":"3b734cf016531a00e10ec9418dafae0983eb8caef8686d7d0c2e0d8eae94981a"} Oct 30 07:34:29.524162 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:29.523772 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-54f2b2-766bc666b6-ljwdv" Oct 30 07:34:29.525648 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:29.525607 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-54f2b2-59f695d955-8fjrt" event={"ID":"4dd49e47-404e-4be7-8e30-26f4ceb2bc1b","Type":"ContainerStarted","Data":"38b10f6f06dec02173a46a80cb52a96444a302df0ad7d9639c03c81bc11b98e6"} Oct 30 07:34:29.525790 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:29.525660 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-54f2b2-59f695d955-8fjrt" event={"ID":"4dd49e47-404e-4be7-8e30-26f4ceb2bc1b","Type":"ContainerStarted","Data":"1f17dce412acaf3c46e2e106071d8eb93f83bea78b9f9cbfa894c4de4d7271f5"} Oct 30 07:34:29.525790 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:29.525691 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-54f2b2-59f695d955-8fjrt" Oct 30 07:34:29.571779 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:29.571730 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-54f2b2-766bc666b6-ljwdv" podStartSLOduration=1.571716149 podStartE2EDuration="1.571716149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:34:29.567892974 +0000 UTC m=+2275.910940939" watchObservedRunningTime="2026-10-30 07:34:29.571716149 +0000 UTC m=+2275.914764113" Oct 30 07:34:29.614472 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:29.614422 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-54f2b2-59f695d955-8fjrt" podStartSLOduration=2.614408451 podStartE2EDuration="2.614408451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:34:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:34:29.611029144 +0000 UTC m=+2275.954077111" watchObservedRunningTime="2026-10-30 07:34:29.614408451 +0000 UTC m=+2275.957456468" Oct 30 07:34:40.537192 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:40.537163 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-54f2b2-59f695d955-8fjrt" Oct 30 07:34:40.537596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:40.537233 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-54f2b2-766bc666b6-ljwdv" Oct 30 07:34:50.134382 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:50.134343 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv"] Oct 30 07:34:50.140693 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:50.140664 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv" Oct 30 07:34:50.160611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:50.159870 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-54f2b2-serving-cert\"" Oct 30 07:34:50.164712 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:50.164681 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv"] Oct 30 07:34:50.231014 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:50.230974 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a568debb-3a25-4cb8-a29c-6cb333299f02-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv\" (UID: \"a568debb-3a25-4cb8-a29c-6cb333299f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv" Oct 30 07:34:50.231014 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:50.231017 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q29zw\" (UniqueName: \"kubernetes.io/projected/a568debb-3a25-4cb8-a29c-6cb333299f02-kube-api-access-q29zw\") pod \"maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv\" (UID: \"a568debb-3a25-4cb8-a29c-6cb333299f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv" Oct 30 07:34:50.231311 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:50.231117 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a568debb-3a25-4cb8-a29c-6cb333299f02-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv\" (UID: \"a568debb-3a25-4cb8-a29c-6cb333299f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv" Oct 30 07:34:50.231360 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:50.231314 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a568debb-3a25-4cb8-a29c-6cb333299f02-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv\" (UID: \"a568debb-3a25-4cb8-a29c-6cb333299f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv" Oct 30 07:34:50.359598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:50.335851 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a568debb-3a25-4cb8-a29c-6cb333299f02-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv\" (UID: \"a568debb-3a25-4cb8-a29c-6cb333299f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv" Oct 30 07:34:50.359598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:50.336120 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a568debb-3a25-4cb8-a29c-6cb333299f02-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv\" (UID: \"a568debb-3a25-4cb8-a29c-6cb333299f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv" Oct 30 07:34:50.359598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:50.336149 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q29zw\" (UniqueName: \"kubernetes.io/projected/a568debb-3a25-4cb8-a29c-6cb333299f02-kube-api-access-q29zw\") pod \"maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv\" (UID: \"a568debb-3a25-4cb8-a29c-6cb333299f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv" Oct 30 07:34:50.359598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:50.336180 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a568debb-3a25-4cb8-a29c-6cb333299f02-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv\" (UID: \"a568debb-3a25-4cb8-a29c-6cb333299f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv" Oct 30 07:34:50.359598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:50.337367 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a568debb-3a25-4cb8-a29c-6cb333299f02-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv\" (UID: \"a568debb-3a25-4cb8-a29c-6cb333299f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv" Oct 30 07:34:50.359598 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:34:50.343030 4951 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-54f2b2-serving-cert: secret "maas-api-e2e-shared-a-w0-54f2b2-serving-cert" not found Oct 30 07:34:50.359598 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:34:50.343125 4951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a568debb-3a25-4cb8-a29c-6cb333299f02-maas-api-tls podName:a568debb-3a25-4cb8-a29c-6cb333299f02 nodeName:}" failed. No retries permitted until 2026-10-30 07:34:50.843101073 +0000 UTC m=+2297.186149018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a568debb-3a25-4cb8-a29c-6cb333299f02-maas-api-tls") pod "maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv" (UID: "a568debb-3a25-4cb8-a29c-6cb333299f02") : secret "maas-api-e2e-shared-a-w0-54f2b2-serving-cert" not found Oct 30 07:34:50.392643 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:50.389379 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q29zw\" (UniqueName: \"kubernetes.io/projected/a568debb-3a25-4cb8-a29c-6cb333299f02-kube-api-access-q29zw\") pod \"maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv\" (UID: \"a568debb-3a25-4cb8-a29c-6cb333299f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv" Oct 30 07:34:50.392643 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:50.390114 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a568debb-3a25-4cb8-a29c-6cb333299f02-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv\" (UID: \"a568debb-3a25-4cb8-a29c-6cb333299f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv" Oct 30 07:34:50.949595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:50.948794 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a568debb-3a25-4cb8-a29c-6cb333299f02-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv\" (UID: \"a568debb-3a25-4cb8-a29c-6cb333299f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv" Oct 30 07:34:50.965739 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:50.965339 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a568debb-3a25-4cb8-a29c-6cb333299f02-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv\" (UID: \"a568debb-3a25-4cb8-a29c-6cb333299f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv" Oct 30 07:34:51.058834 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:51.058797 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv" Oct 30 07:34:51.368473 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:34:51.368441 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda568debb_3a25_4cb8_a29c_6cb333299f02.slice/crio-5b53a93d9b8c7530e428ed60ebae776d65c3477b324f9e39acd45588cb9dfa4a WatchSource:0}: Error finding container 5b53a93d9b8c7530e428ed60ebae776d65c3477b324f9e39acd45588cb9dfa4a: Status 404 returned error can't find the container with id 5b53a93d9b8c7530e428ed60ebae776d65c3477b324f9e39acd45588cb9dfa4a Oct 30 07:34:51.369634 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:51.369604 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv"] Oct 30 07:34:51.699471 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:51.699444 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv" event={"ID":"a568debb-3a25-4cb8-a29c-6cb333299f02","Type":"ContainerStarted","Data":"5b53a93d9b8c7530e428ed60ebae776d65c3477b324f9e39acd45588cb9dfa4a"} Oct 30 07:34:52.707214 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:52.707174 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv" event={"ID":"a568debb-3a25-4cb8-a29c-6cb333299f02","Type":"ContainerStarted","Data":"33c0248966b61e21e362840c835a9885ca5d86ffd671bf84178ba9aff78f9c72"} Oct 30 07:34:52.707668 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:52.707261 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv" Oct 30 07:34:52.756115 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:52.756052 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv" podStartSLOduration=2.464964968 podStartE2EDuration="2.756036856s" totalImagesPullingTime="291.071888ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:34:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:34:52.75305876 +0000 UTC m=+2299.096106723" watchObservedRunningTime="2026-10-30 07:34:52.756036856 +0000 UTC m=+2299.099084821" Oct 30 07:34:53.024689 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:34:53.024649 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:34:55.779763 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:55.779721 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7599-a46a6d"] Oct 30 07:34:55.784989 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:55.784961 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-a46a6d" Oct 30 07:34:55.802365 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:55.802324 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7599-a46a6d"] Oct 30 07:34:55.812175 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:55.812136 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5z8jp\"" Oct 30 07:34:55.881801 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:55.881768 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4752\" (UniqueName: \"kubernetes.io/projected/369d493f-9354-4b8f-830e-439276a3b2ca-kube-api-access-v4752\") pod \"test-curl-7599-a46a6d\" (UID: \"369d493f-9354-4b8f-830e-439276a3b2ca\") " pod="opendatahub/test-curl-7599-a46a6d" Oct 30 07:34:55.983208 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:55.983168 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4752\" (UniqueName: \"kubernetes.io/projected/369d493f-9354-4b8f-830e-439276a3b2ca-kube-api-access-v4752\") pod \"test-curl-7599-a46a6d\" (UID: \"369d493f-9354-4b8f-830e-439276a3b2ca\") " pod="opendatahub/test-curl-7599-a46a6d" Oct 30 07:34:56.008651 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:56.008616 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4752\" (UniqueName: \"kubernetes.io/projected/369d493f-9354-4b8f-830e-439276a3b2ca-kube-api-access-v4752\") pod \"test-curl-7599-a46a6d\" (UID: \"369d493f-9354-4b8f-830e-439276a3b2ca\") " pod="opendatahub/test-curl-7599-a46a6d" Oct 30 07:34:56.098689 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:56.098524 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-a46a6d" Oct 30 07:34:56.418630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:56.418539 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7599-a46a6d"] Oct 30 07:34:56.746592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:56.746481 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7599-a46a6d" event={"ID":"369d493f-9354-4b8f-830e-439276a3b2ca","Type":"ContainerStarted","Data":"e6432e2c72be340e9c046a71a969cc7d4bdfcf40579d65862b2a889604d75081"} Oct 30 07:34:56.746592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:56.746535 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7599-a46a6d" event={"ID":"369d493f-9354-4b8f-830e-439276a3b2ca","Type":"ContainerStarted","Data":"5d9ef3f80959bbe616c38853012f06bb92d9849bcc8f033fe3d186ac9010c5ab"} Oct 30 07:34:56.793472 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:56.793418 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7599-a46a6d" podStartSLOduration=1.793403274 podStartE2EDuration="1.793403274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:34:56.792342427 +0000 UTC m=+2303.135390417" watchObservedRunningTime="2026-10-30 07:34:56.793403274 +0000 UTC m=+2303.136451239" Oct 30 07:34:57.623309 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:57.622387 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7599-a46a6d"] Oct 30 07:34:57.627482 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:57.627449 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7599-a46a6d"] Oct 30 07:34:57.784700 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:57.784664 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7599-954174"] Oct 30 07:34:57.785688 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:57.785667 4951 cpu_manager.go:414] "containerMap: removing container" podUID="369d493f-9354-4b8f-830e-439276a3b2ca" containerName="test-curl-7599-a46a6d" Oct 30 07:34:57.785739 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:57.785692 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="369d493f-9354-4b8f-830e-439276a3b2ca" containerName="test-curl-7599-a46a6d" Oct 30 07:34:57.785840 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:57.785829 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="369d493f-9354-4b8f-830e-439276a3b2ca" containerName="test-curl-7599-a46a6d" Oct 30 07:34:57.789577 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:57.789545 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-954174" Oct 30 07:34:57.799708 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:57.799680 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7599-954174"] Oct 30 07:34:57.809968 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:57.809934 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cs2m\" (UniqueName: \"kubernetes.io/projected/2db41d15-cac6-45f5-8b7c-2869895562a7-kube-api-access-6cs2m\") pod \"test-curl-7599-954174\" (UID: \"2db41d15-cac6-45f5-8b7c-2869895562a7\") " pod="opendatahub/test-curl-7599-954174" Oct 30 07:34:57.911239 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:57.911144 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cs2m\" (UniqueName: \"kubernetes.io/projected/2db41d15-cac6-45f5-8b7c-2869895562a7-kube-api-access-6cs2m\") pod \"test-curl-7599-954174\" (UID: \"2db41d15-cac6-45f5-8b7c-2869895562a7\") " pod="opendatahub/test-curl-7599-954174" Oct 30 07:34:57.936485 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:57.936445 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cs2m\" (UniqueName: \"kubernetes.io/projected/2db41d15-cac6-45f5-8b7c-2869895562a7-kube-api-access-6cs2m\") pod \"test-curl-7599-954174\" (UID: \"2db41d15-cac6-45f5-8b7c-2869895562a7\") " pod="opendatahub/test-curl-7599-954174" Oct 30 07:34:58.105287 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:58.105241 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-954174" Oct 30 07:34:58.411180 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:34:58.411146 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2db41d15_cac6_45f5_8b7c_2869895562a7.slice/crio-f803ef748dedeed5dd3a55d2260fd2d6a53ad74dc718fd3381d66ad37c0c9114 WatchSource:0}: Error finding container f803ef748dedeed5dd3a55d2260fd2d6a53ad74dc718fd3381d66ad37c0c9114: Status 404 returned error can't find the container with id f803ef748dedeed5dd3a55d2260fd2d6a53ad74dc718fd3381d66ad37c0c9114 Oct 30 07:34:58.411852 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:58.411792 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7599-954174"] Oct 30 07:34:58.720492 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:58.720422 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv" Oct 30 07:34:58.761663 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:58.761627 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7599-954174" event={"ID":"2db41d15-cac6-45f5-8b7c-2869895562a7","Type":"ContainerStarted","Data":"785f2c1c9964a3ad8d305e525d033871c0ac9b17d2c8ae81db284f3f3d103dc9"} Oct 30 07:34:58.761932 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:58.761674 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7599-954174" event={"ID":"2db41d15-cac6-45f5-8b7c-2869895562a7","Type":"ContainerStarted","Data":"f803ef748dedeed5dd3a55d2260fd2d6a53ad74dc718fd3381d66ad37c0c9114"} Oct 30 07:34:58.761932 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:58.761711 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7599-a46a6d" podUID="369d493f-9354-4b8f-830e-439276a3b2ca" containerName="test-curl-7599-a46a6d" containerID="cri-o://e6432e2c72be340e9c046a71a969cc7d4bdfcf40579d65862b2a889604d75081" gracePeriod=2 Oct 30 07:34:58.858082 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:58.858008 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7599-954174" podStartSLOduration=1.857988178 podStartE2EDuration="1.857988178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:34:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:34:58.857496217 +0000 UTC m=+2305.200544204" watchObservedRunningTime="2026-10-30 07:34:58.857988178 +0000 UTC m=+2305.201036144" Oct 30 07:34:59.821094 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:59.821062 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7599-954174"] Oct 30 07:34:59.825916 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:59.825882 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7599-954174"] Oct 30 07:34:59.968347 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:59.968302 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7599-d5fe4f"] Oct 30 07:34:59.969000 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:59.968984 4951 cpu_manager.go:414] "containerMap: removing container" podUID="2db41d15-cac6-45f5-8b7c-2869895562a7" containerName="test-curl-7599-954174" Oct 30 07:34:59.969073 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:59.969001 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2db41d15-cac6-45f5-8b7c-2869895562a7" containerName="test-curl-7599-954174" Oct 30 07:34:59.969154 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:59.969144 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="2db41d15-cac6-45f5-8b7c-2869895562a7" containerName="test-curl-7599-954174" Oct 30 07:34:59.972546 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:59.972521 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-d5fe4f" Oct 30 07:34:59.992975 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:34:59.992951 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7599-d5fe4f"] Oct 30 07:35:00.057484 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:00.057445 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lj4z\" (UniqueName: \"kubernetes.io/projected/44e8dbfd-bdbb-45e8-9202-a37ea13973b9-kube-api-access-9lj4z\") pod \"test-curl-7599-d5fe4f\" (UID: \"44e8dbfd-bdbb-45e8-9202-a37ea13973b9\") " pod="opendatahub/test-curl-7599-d5fe4f" Oct 30 07:35:00.159405 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:00.159317 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lj4z\" (UniqueName: \"kubernetes.io/projected/44e8dbfd-bdbb-45e8-9202-a37ea13973b9-kube-api-access-9lj4z\") pod \"test-curl-7599-d5fe4f\" (UID: \"44e8dbfd-bdbb-45e8-9202-a37ea13973b9\") " pod="opendatahub/test-curl-7599-d5fe4f" Oct 30 07:35:00.184623 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:00.184544 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lj4z\" (UniqueName: \"kubernetes.io/projected/44e8dbfd-bdbb-45e8-9202-a37ea13973b9-kube-api-access-9lj4z\") pod \"test-curl-7599-d5fe4f\" (UID: \"44e8dbfd-bdbb-45e8-9202-a37ea13973b9\") " pod="opendatahub/test-curl-7599-d5fe4f" Oct 30 07:35:00.285476 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:00.285437 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-d5fe4f" Oct 30 07:35:00.593455 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:00.593424 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7599-d5fe4f"] Oct 30 07:35:00.593627 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:35:00.593416 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44e8dbfd_bdbb_45e8_9202_a37ea13973b9.slice/crio-28e0ac96d098c8286eda6ef45729b71022eaadb209b1320c37a7550b28185f0e WatchSource:0}: Error finding container 28e0ac96d098c8286eda6ef45729b71022eaadb209b1320c37a7550b28185f0e: Status 404 returned error can't find the container with id 28e0ac96d098c8286eda6ef45729b71022eaadb209b1320c37a7550b28185f0e Oct 30 07:35:00.777321 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:00.777289 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7599-d5fe4f" event={"ID":"44e8dbfd-bdbb-45e8-9202-a37ea13973b9","Type":"ContainerStarted","Data":"27c83fdbe9abd1f6a691d9a37918fb0eb9b9d08ab933317a1bdd9c656ab43f8c"} Oct 30 07:35:00.777455 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:00.777346 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7599-d5fe4f" event={"ID":"44e8dbfd-bdbb-45e8-9202-a37ea13973b9","Type":"ContainerStarted","Data":"28e0ac96d098c8286eda6ef45729b71022eaadb209b1320c37a7550b28185f0e"} Oct 30 07:35:00.777455 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:00.777401 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7599-954174" podUID="2db41d15-cac6-45f5-8b7c-2869895562a7" containerName="test-curl-7599-954174" containerID="cri-o://785f2c1c9964a3ad8d305e525d033871c0ac9b17d2c8ae81db284f3f3d103dc9" gracePeriod=2 Oct 30 07:35:00.794847 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:35:00.794810 4951 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7599-954174\" is forbidden: User \"system:node:ip-10-0-132-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-173.ec2.internal' and this object" podUID="2db41d15-cac6-45f5-8b7c-2869895562a7" pod="opendatahub/test-curl-7599-954174" Oct 30 07:35:00.838405 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:00.837999 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7599-d5fe4f" podStartSLOduration=1.8379844520000002 podStartE2EDuration="1.837984452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:35:00.835786674 +0000 UTC m=+2307.178834640" watchObservedRunningTime="2026-10-30 07:35:00.837984452 +0000 UTC m=+2307.181032416" Oct 30 07:35:00.851558 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:35:00.851461 4951 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7599-954174\" is forbidden: User \"system:node:ip-10-0-132-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-173.ec2.internal' and this object" podUID="2db41d15-cac6-45f5-8b7c-2869895562a7" pod="opendatahub/test-curl-7599-954174" Oct 30 07:35:01.030722 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:01.030693 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-a46a6d" Oct 30 07:35:01.045887 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:35:01.045848 4951 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7599-954174\" is forbidden: User \"system:node:ip-10-0-132-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-173.ec2.internal' and this object" podUID="2db41d15-cac6-45f5-8b7c-2869895562a7" pod="opendatahub/test-curl-7599-954174" Oct 30 07:35:01.179147 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:01.179070 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/369d493f-9354-4b8f-830e-439276a3b2ca-kube-api-access-v4752\" (UniqueName: \"kubernetes.io/projected/369d493f-9354-4b8f-830e-439276a3b2ca-kube-api-access-v4752\") pod \"369d493f-9354-4b8f-830e-439276a3b2ca\" (UID: \"369d493f-9354-4b8f-830e-439276a3b2ca\") " Oct 30 07:35:01.181191 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:01.181162 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/369d493f-9354-4b8f-830e-439276a3b2ca-kube-api-access-v4752" pod "369d493f-9354-4b8f-830e-439276a3b2ca" (UID: "369d493f-9354-4b8f-830e-439276a3b2ca"). InnerVolumeSpecName "kube-api-access-v4752". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:35:01.280997 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:01.280960 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4752\" (UniqueName: \"kubernetes.io/projected/369d493f-9354-4b8f-830e-439276a3b2ca-kube-api-access-v4752\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:35:01.689217 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:01.689180 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7599-d5fe4f"] Oct 30 07:35:01.694185 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:01.694158 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7599-d5fe4f"] Oct 30 07:35:01.772002 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:01.771969 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="369d493f-9354-4b8f-830e-439276a3b2ca" path="/var/lib/kubelet/pods/369d493f-9354-4b8f-830e-439276a3b2ca/volumes" Oct 30 07:35:01.784257 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:01.784231 4951 generic.go:356] "Generic (PLEG): container finished" podID="369d493f-9354-4b8f-830e-439276a3b2ca" containerID="e6432e2c72be340e9c046a71a969cc7d4bdfcf40579d65862b2a889604d75081" exitCode=137 Oct 30 07:35:01.784376 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:01.784287 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-a46a6d" Oct 30 07:35:01.784376 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:01.784326 4951 scope.go:122] "RemoveContainer" containerID="e6432e2c72be340e9c046a71a969cc7d4bdfcf40579d65862b2a889604d75081" Oct 30 07:35:01.794306 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:01.794287 4951 scope.go:122] "RemoveContainer" containerID="e6432e2c72be340e9c046a71a969cc7d4bdfcf40579d65862b2a889604d75081" Oct 30 07:35:01.794587 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:35:01.794550 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6432e2c72be340e9c046a71a969cc7d4bdfcf40579d65862b2a889604d75081\": container with ID starting with e6432e2c72be340e9c046a71a969cc7d4bdfcf40579d65862b2a889604d75081 not found: ID does not exist" containerID="e6432e2c72be340e9c046a71a969cc7d4bdfcf40579d65862b2a889604d75081" Oct 30 07:35:01.794669 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:01.794595 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6432e2c72be340e9c046a71a969cc7d4bdfcf40579d65862b2a889604d75081"} err="failed to get container status \"e6432e2c72be340e9c046a71a969cc7d4bdfcf40579d65862b2a889604d75081\": rpc error: code = NotFound desc = could not find container \"e6432e2c72be340e9c046a71a969cc7d4bdfcf40579d65862b2a889604d75081\": container with ID starting with e6432e2c72be340e9c046a71a969cc7d4bdfcf40579d65862b2a889604d75081 not found: ID does not exist" Oct 30 07:35:01.843078 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:01.843033 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7599-ee54d5"] Oct 30 07:35:01.843749 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:01.843733 4951 cpu_manager.go:414] "containerMap: removing container" podUID="44e8dbfd-bdbb-45e8-9202-a37ea13973b9" containerName="test-curl-7599-d5fe4f" Oct 30 07:35:01.843811 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:01.843751 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44e8dbfd-bdbb-45e8-9202-a37ea13973b9" containerName="test-curl-7599-d5fe4f" Oct 30 07:35:01.843900 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:01.843891 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="44e8dbfd-bdbb-45e8-9202-a37ea13973b9" containerName="test-curl-7599-d5fe4f" Oct 30 07:35:01.849129 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:01.849093 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-ee54d5" Oct 30 07:35:01.866222 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:01.866183 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7599-ee54d5"] Oct 30 07:35:01.990771 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:01.990682 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dddlq\" (UniqueName: \"kubernetes.io/projected/ea10e9de-9c00-4585-ab2d-811d54bad0bd-kube-api-access-dddlq\") pod \"test-curl-7599-ee54d5\" (UID: \"ea10e9de-9c00-4585-ab2d-811d54bad0bd\") " pod="opendatahub/test-curl-7599-ee54d5" Oct 30 07:35:02.092212 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:02.092174 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dddlq\" (UniqueName: \"kubernetes.io/projected/ea10e9de-9c00-4585-ab2d-811d54bad0bd-kube-api-access-dddlq\") pod \"test-curl-7599-ee54d5\" (UID: \"ea10e9de-9c00-4585-ab2d-811d54bad0bd\") " pod="opendatahub/test-curl-7599-ee54d5" Oct 30 07:35:02.118002 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:02.117964 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dddlq\" (UniqueName: \"kubernetes.io/projected/ea10e9de-9c00-4585-ab2d-811d54bad0bd-kube-api-access-dddlq\") pod \"test-curl-7599-ee54d5\" (UID: \"ea10e9de-9c00-4585-ab2d-811d54bad0bd\") " pod="opendatahub/test-curl-7599-ee54d5" Oct 30 07:35:02.163187 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:02.163149 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-ee54d5" Oct 30 07:35:02.475753 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:35:02.475720 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea10e9de_9c00_4585_ab2d_811d54bad0bd.slice/crio-76ed02c55ce9ead4a51f9178db36d42ad8cd05c9eedd53e54460a891f578779e WatchSource:0}: Error finding container 76ed02c55ce9ead4a51f9178db36d42ad8cd05c9eedd53e54460a891f578779e: Status 404 returned error can't find the container with id 76ed02c55ce9ead4a51f9178db36d42ad8cd05c9eedd53e54460a891f578779e Oct 30 07:35:02.476195 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:02.476171 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7599-ee54d5"] Oct 30 07:35:02.795511 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:02.795477 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7599-ee54d5" event={"ID":"ea10e9de-9c00-4585-ab2d-811d54bad0bd","Type":"ContainerStarted","Data":"54c8aecd9bed3e87b87ac03cdf83161710e2733b4c59703c5ee49dbbad804ff3"} Oct 30 07:35:02.795727 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:02.795523 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7599-ee54d5" event={"ID":"ea10e9de-9c00-4585-ab2d-811d54bad0bd","Type":"ContainerStarted","Data":"76ed02c55ce9ead4a51f9178db36d42ad8cd05c9eedd53e54460a891f578779e"} Oct 30 07:35:02.795727 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:02.795595 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7599-d5fe4f" podUID="44e8dbfd-bdbb-45e8-9202-a37ea13973b9" containerName="test-curl-7599-d5fe4f" containerID="cri-o://27c83fdbe9abd1f6a691d9a37918fb0eb9b9d08ab933317a1bdd9c656ab43f8c" gracePeriod=2 Oct 30 07:35:02.839797 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:02.839745 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7599-ee54d5" podStartSLOduration=1.83972756 podStartE2EDuration="1.83972756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:35:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:35:02.837025043 +0000 UTC m=+2309.180073008" watchObservedRunningTime="2026-10-30 07:35:02.83972756 +0000 UTC m=+2309.182775526" Oct 30 07:35:02.989998 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:02.989969 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-954174" Oct 30 07:35:03.107348 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:03.107266 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2db41d15-cac6-45f5-8b7c-2869895562a7-kube-api-access-6cs2m\" (UniqueName: \"kubernetes.io/projected/2db41d15-cac6-45f5-8b7c-2869895562a7-kube-api-access-6cs2m\") pod \"2db41d15-cac6-45f5-8b7c-2869895562a7\" (UID: \"2db41d15-cac6-45f5-8b7c-2869895562a7\") " Oct 30 07:35:03.109491 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:03.109464 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db41d15-cac6-45f5-8b7c-2869895562a7-kube-api-access-6cs2m" pod "2db41d15-cac6-45f5-8b7c-2869895562a7" (UID: "2db41d15-cac6-45f5-8b7c-2869895562a7"). InnerVolumeSpecName "kube-api-access-6cs2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:35:03.209130 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:03.209102 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cs2m\" (UniqueName: \"kubernetes.io/projected/2db41d15-cac6-45f5-8b7c-2869895562a7-kube-api-access-6cs2m\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:35:03.713947 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:03.713913 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7599-ee54d5"] Oct 30 07:35:03.722358 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:03.722326 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7599-ee54d5"] Oct 30 07:35:03.783481 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:03.782105 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2db41d15-cac6-45f5-8b7c-2869895562a7" path="/var/lib/kubelet/pods/2db41d15-cac6-45f5-8b7c-2869895562a7/volumes" Oct 30 07:35:03.804587 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:03.804543 4951 generic.go:356] "Generic (PLEG): container finished" podID="2db41d15-cac6-45f5-8b7c-2869895562a7" containerID="785f2c1c9964a3ad8d305e525d033871c0ac9b17d2c8ae81db284f3f3d103dc9" exitCode=137 Oct 30 07:35:03.804781 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:03.804616 4951 scope.go:122] "RemoveContainer" containerID="785f2c1c9964a3ad8d305e525d033871c0ac9b17d2c8ae81db284f3f3d103dc9" Oct 30 07:35:03.804781 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:03.804637 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-954174" Oct 30 07:35:03.815197 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:03.815170 4951 scope.go:122] "RemoveContainer" containerID="785f2c1c9964a3ad8d305e525d033871c0ac9b17d2c8ae81db284f3f3d103dc9" Oct 30 07:35:03.815524 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:35:03.815502 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"785f2c1c9964a3ad8d305e525d033871c0ac9b17d2c8ae81db284f3f3d103dc9\": container with ID starting with 785f2c1c9964a3ad8d305e525d033871c0ac9b17d2c8ae81db284f3f3d103dc9 not found: ID does not exist" containerID="785f2c1c9964a3ad8d305e525d033871c0ac9b17d2c8ae81db284f3f3d103dc9" Oct 30 07:35:03.815634 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:03.815533 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"785f2c1c9964a3ad8d305e525d033871c0ac9b17d2c8ae81db284f3f3d103dc9"} err="failed to get container status \"785f2c1c9964a3ad8d305e525d033871c0ac9b17d2c8ae81db284f3f3d103dc9\": rpc error: code = NotFound desc = could not find container \"785f2c1c9964a3ad8d305e525d033871c0ac9b17d2c8ae81db284f3f3d103dc9\": container with ID starting with 785f2c1c9964a3ad8d305e525d033871c0ac9b17d2c8ae81db284f3f3d103dc9 not found: ID does not exist" Oct 30 07:35:04.813668 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:04.813602 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7599-ee54d5" podUID="ea10e9de-9c00-4585-ab2d-811d54bad0bd" containerName="test-curl-7599-ee54d5" containerID="cri-o://54c8aecd9bed3e87b87ac03cdf83161710e2733b4c59703c5ee49dbbad804ff3" gracePeriod=2 Oct 30 07:35:05.043259 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:05.043230 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-d5fe4f" Oct 30 07:35:05.158128 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:05.152483 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44e8dbfd-bdbb-45e8-9202-a37ea13973b9-kube-api-access-9lj4z\" (UniqueName: \"kubernetes.io/projected/44e8dbfd-bdbb-45e8-9202-a37ea13973b9-kube-api-access-9lj4z\") pod \"44e8dbfd-bdbb-45e8-9202-a37ea13973b9\" (UID: \"44e8dbfd-bdbb-45e8-9202-a37ea13973b9\") " Oct 30 07:35:05.170065 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:05.162175 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44e8dbfd-bdbb-45e8-9202-a37ea13973b9-kube-api-access-9lj4z" pod "44e8dbfd-bdbb-45e8-9202-a37ea13973b9" (UID: "44e8dbfd-bdbb-45e8-9202-a37ea13973b9"). InnerVolumeSpecName "kube-api-access-9lj4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:35:05.267595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:05.267533 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lj4z\" (UniqueName: \"kubernetes.io/projected/44e8dbfd-bdbb-45e8-9202-a37ea13973b9-kube-api-access-9lj4z\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:35:05.705010 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:05.704972 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5"] Oct 30 07:35:05.706036 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:05.706001 4951 cpu_manager.go:414] "containerMap: removing container" podUID="ea10e9de-9c00-4585-ab2d-811d54bad0bd" containerName="test-curl-7599-ee54d5" Oct 30 07:35:05.706036 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:05.706028 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea10e9de-9c00-4585-ab2d-811d54bad0bd" containerName="test-curl-7599-ee54d5" Oct 30 07:35:05.706253 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:05.706233 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea10e9de-9c00-4585-ab2d-811d54bad0bd" containerName="test-curl-7599-ee54d5" Oct 30 07:35:05.710560 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:05.710533 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:35:05.729757 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:05.729689 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-1e0880-openshift-default-dockercfg-4g54q\"" Oct 30 07:35:05.754407 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:05.754375 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5"] Oct 30 07:35:05.776444 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:05.776413 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44e8dbfd-bdbb-45e8-9202-a37ea13973b9" path="/var/lib/kubelet/pods/44e8dbfd-bdbb-45e8-9202-a37ea13973b9/volumes" Oct 30 07:35:05.918596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:05.918061 4951 generic.go:356] "Generic (PLEG): container finished" podID="44e8dbfd-bdbb-45e8-9202-a37ea13973b9" containerID="27c83fdbe9abd1f6a691d9a37918fb0eb9b9d08ab933317a1bdd9c656ab43f8c" exitCode=137 Oct 30 07:35:05.938597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:05.920713 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-d5fe4f" Oct 30 07:35:05.938597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:05.928792 4951 scope.go:122] "RemoveContainer" containerID="27c83fdbe9abd1f6a691d9a37918fb0eb9b9d08ab933317a1bdd9c656ab43f8c" Oct 30 07:35:05.938597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:05.934604 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmm9w\" (UniqueName: \"kubernetes.io/projected/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-kube-api-access-pmm9w\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:35:05.938597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:05.934686 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istio-envoy\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:35:05.938597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:05.934823 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istio-token\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:35:05.938597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:05.934860 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istio-podinfo\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:35:05.938597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:05.934927 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-workload-certs\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:35:05.938597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:05.934972 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istio-data\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:35:05.938597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:05.935009 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istiod-ca-cert\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:35:05.938597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:05.935050 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-workload-socket\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:35:05.938597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:05.935178 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-credential-socket\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:35:06.018098 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:06.017971 4951 scope.go:122] "RemoveContainer" containerID="27c83fdbe9abd1f6a691d9a37918fb0eb9b9d08ab933317a1bdd9c656ab43f8c" Oct 30 07:35:06.024604 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:35:06.018520 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c83fdbe9abd1f6a691d9a37918fb0eb9b9d08ab933317a1bdd9c656ab43f8c\": container with ID starting with 27c83fdbe9abd1f6a691d9a37918fb0eb9b9d08ab933317a1bdd9c656ab43f8c not found: ID does not exist" containerID="27c83fdbe9abd1f6a691d9a37918fb0eb9b9d08ab933317a1bdd9c656ab43f8c" Oct 30 07:35:06.024865 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:06.018555 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c83fdbe9abd1f6a691d9a37918fb0eb9b9d08ab933317a1bdd9c656ab43f8c"} err="failed to get container status \"27c83fdbe9abd1f6a691d9a37918fb0eb9b9d08ab933317a1bdd9c656ab43f8c\": rpc error: code = NotFound desc = could not find container \"27c83fdbe9abd1f6a691d9a37918fb0eb9b9d08ab933317a1bdd9c656ab43f8c\": container with ID starting with 27c83fdbe9abd1f6a691d9a37918fb0eb9b9d08ab933317a1bdd9c656ab43f8c not found: ID does not exist" Oct 30 07:35:06.038626 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:06.037691 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-credential-socket\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:35:06.038626 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:06.037784 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmm9w\" (UniqueName: \"kubernetes.io/projected/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-kube-api-access-pmm9w\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:35:06.038626 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:06.037836 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istio-envoy\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:35:06.038626 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:06.037946 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istio-token\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:35:06.038626 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:06.037983 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istio-podinfo\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:35:06.038626 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:06.038041 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-workload-certs\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:35:06.038626 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:06.038084 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istio-data\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:35:06.038626 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:06.038122 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istiod-ca-cert\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:35:06.038626 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:06.038161 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-workload-socket\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:35:06.039817 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:06.039789 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-workload-socket\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:35:06.041348 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:06.040192 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-credential-socket\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:35:06.041472 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:06.040957 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-workload-certs\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:35:06.057751 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:06.050954 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istio-data\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:35:06.058148 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:06.051538 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istiod-ca-cert\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:35:06.061443 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:06.060745 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istio-podinfo\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:35:06.077207 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:06.066925 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istio-envoy\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:35:06.086639 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:06.080541 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istio-token\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:35:06.092646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:06.089693 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmm9w\" (UniqueName: \"kubernetes.io/projected/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-kube-api-access-pmm9w\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:35:06.326047 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:06.325994 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:35:06.655623 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:06.653748 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5"] Oct 30 07:35:06.660549 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:35:06.660480 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b29cf13_7e6a_42d3_9a18_bcd9e9fe48e3.slice/crio-b582f9ff8c31624554f64b79f6d0f851c2468eb102bc06c797694be7b282eb41 WatchSource:0}: Error finding container b582f9ff8c31624554f64b79f6d0f851c2468eb102bc06c797694be7b282eb41: Status 404 returned error can't find the container with id b582f9ff8c31624554f64b79f6d0f851c2468eb102bc06c797694be7b282eb41 Oct 30 07:35:06.663232 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:06.663194 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:35:06.663337 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:06.663286 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:35:06.663337 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:06.663322 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:35:06.937411 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:06.937338 4951 generic.go:356] "Generic (PLEG): container finished" podID="ea10e9de-9c00-4585-ab2d-811d54bad0bd" containerID="54c8aecd9bed3e87b87ac03cdf83161710e2733b4c59703c5ee49dbbad804ff3" exitCode=137 Oct 30 07:35:06.939684 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:06.939657 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" event={"ID":"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3","Type":"ContainerStarted","Data":"5c6a8eec409b079ca0e33fab991eb8bbaae4ac44bce0f48415332c5b33d0dbc2"} Oct 30 07:35:06.939896 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:06.939878 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" event={"ID":"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3","Type":"ContainerStarted","Data":"b582f9ff8c31624554f64b79f6d0f851c2468eb102bc06c797694be7b282eb41"} Oct 30 07:35:07.069874 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:07.069844 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-ee54d5" Oct 30 07:35:07.156597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:07.155344 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea10e9de-9c00-4585-ab2d-811d54bad0bd-kube-api-access-dddlq\" (UniqueName: \"kubernetes.io/projected/ea10e9de-9c00-4585-ab2d-811d54bad0bd-kube-api-access-dddlq\") pod \"ea10e9de-9c00-4585-ab2d-811d54bad0bd\" (UID: \"ea10e9de-9c00-4585-ab2d-811d54bad0bd\") " Oct 30 07:35:07.169595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:07.166368 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea10e9de-9c00-4585-ab2d-811d54bad0bd-kube-api-access-dddlq" pod "ea10e9de-9c00-4585-ab2d-811d54bad0bd" (UID: "ea10e9de-9c00-4585-ab2d-811d54bad0bd"). InnerVolumeSpecName "kube-api-access-dddlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:35:07.257275 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:07.257239 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dddlq\" (UniqueName: \"kubernetes.io/projected/ea10e9de-9c00-4585-ab2d-811d54bad0bd-kube-api-access-dddlq\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:35:07.327129 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:07.327097 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:35:07.333025 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:07.332999 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:35:07.386778 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:07.386729 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" podStartSLOduration=2.386714624 podStartE2EDuration="2.386714624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:35:07.002991761 +0000 UTC m=+2313.346039752" watchObservedRunningTime="2026-10-30 07:35:07.386714624 +0000 UTC m=+2313.729762636" Oct 30 07:35:07.773803 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:07.773766 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea10e9de-9c00-4585-ab2d-811d54bad0bd" path="/var/lib/kubelet/pods/ea10e9de-9c00-4585-ab2d-811d54bad0bd/volumes" Oct 30 07:35:07.950032 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:07.950002 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-ee54d5" Oct 30 07:35:07.950546 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:07.950053 4951 scope.go:122] "RemoveContainer" containerID="54c8aecd9bed3e87b87ac03cdf83161710e2733b4c59703c5ee49dbbad804ff3" Oct 30 07:35:07.950546 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:07.950470 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:35:07.951837 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:07.951819 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:35:11.209420 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:11.209369 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1e0880-6946fccf5f-2rcnf"] Oct 30 07:35:11.215279 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:11.215253 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1e0880-6946fccf5f-2rcnf" Oct 30 07:35:11.235053 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:11.235019 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1e0880-6946fccf5f-2rcnf"] Oct 30 07:35:11.243137 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:11.243110 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-1e0880\"" Oct 30 07:35:11.243137 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:11.243119 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-1e0880-dockercfg-x2fvm\"" Oct 30 07:35:11.352598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:11.336649 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ba04d1-bbfe-4ed3-b0b2-55d554a025fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-1e0880-6946fccf5f-2rcnf\" (UID: \"64ba04d1-bbfe-4ed3-b0b2-55d554a025fb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1e0880-6946fccf5f-2rcnf" Oct 30 07:35:11.376598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:11.353088 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26swz\" (UniqueName: \"kubernetes.io/projected/64ba04d1-bbfe-4ed3-b0b2-55d554a025fb-kube-api-access-26swz\") pod \"payload-pre-processing-e2e-extmod-a-1e0880-6946fccf5f-2rcnf\" (UID: \"64ba04d1-bbfe-4ed3-b0b2-55d554a025fb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1e0880-6946fccf5f-2rcnf" Oct 30 07:35:11.403604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:11.347322 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1e0880-79484bb6bd-5vfjq"] Oct 30 07:35:11.495686 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:11.487212 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ba04d1-bbfe-4ed3-b0b2-55d554a025fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-1e0880-6946fccf5f-2rcnf\" (UID: \"64ba04d1-bbfe-4ed3-b0b2-55d554a025fb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1e0880-6946fccf5f-2rcnf" Oct 30 07:35:11.495686 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:11.487342 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26swz\" (UniqueName: \"kubernetes.io/projected/64ba04d1-bbfe-4ed3-b0b2-55d554a025fb-kube-api-access-26swz\") pod \"payload-pre-processing-e2e-extmod-a-1e0880-6946fccf5f-2rcnf\" (UID: \"64ba04d1-bbfe-4ed3-b0b2-55d554a025fb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1e0880-6946fccf5f-2rcnf" Oct 30 07:35:11.504044 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:11.502874 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ba04d1-bbfe-4ed3-b0b2-55d554a025fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-1e0880-6946fccf5f-2rcnf\" (UID: \"64ba04d1-bbfe-4ed3-b0b2-55d554a025fb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1e0880-6946fccf5f-2rcnf" Oct 30 07:35:11.513509 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:11.512749 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1e0880-79484bb6bd-5vfjq"] Oct 30 07:35:11.513509 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:11.512863 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1e0880-79484bb6bd-5vfjq" Oct 30 07:35:11.518348 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:11.517842 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26swz\" (UniqueName: \"kubernetes.io/projected/64ba04d1-bbfe-4ed3-b0b2-55d554a025fb-kube-api-access-26swz\") pod \"payload-pre-processing-e2e-extmod-a-1e0880-6946fccf5f-2rcnf\" (UID: \"64ba04d1-bbfe-4ed3-b0b2-55d554a025fb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1e0880-6946fccf5f-2rcnf" Oct 30 07:35:11.533516 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:11.533484 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1e0880-6946fccf5f-2rcnf" Oct 30 07:35:11.597411 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:11.591189 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95jlm\" (UniqueName: \"kubernetes.io/projected/33977737-ec70-47e5-a689-4c471c77c42e-kube-api-access-95jlm\") pod \"payload-processing-e2e-extmod-a-1e0880-79484bb6bd-5vfjq\" (UID: \"33977737-ec70-47e5-a689-4c471c77c42e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1e0880-79484bb6bd-5vfjq" Oct 30 07:35:11.597411 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:11.591332 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33977737-ec70-47e5-a689-4c471c77c42e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-1e0880-79484bb6bd-5vfjq\" (UID: \"33977737-ec70-47e5-a689-4c471c77c42e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1e0880-79484bb6bd-5vfjq" Oct 30 07:35:11.692813 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:11.692774 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95jlm\" (UniqueName: \"kubernetes.io/projected/33977737-ec70-47e5-a689-4c471c77c42e-kube-api-access-95jlm\") pod \"payload-processing-e2e-extmod-a-1e0880-79484bb6bd-5vfjq\" (UID: \"33977737-ec70-47e5-a689-4c471c77c42e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1e0880-79484bb6bd-5vfjq" Oct 30 07:35:11.692977 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:11.692889 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33977737-ec70-47e5-a689-4c471c77c42e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-1e0880-79484bb6bd-5vfjq\" (UID: \"33977737-ec70-47e5-a689-4c471c77c42e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1e0880-79484bb6bd-5vfjq" Oct 30 07:35:11.693854 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:11.693806 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33977737-ec70-47e5-a689-4c471c77c42e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-1e0880-79484bb6bd-5vfjq\" (UID: \"33977737-ec70-47e5-a689-4c471c77c42e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1e0880-79484bb6bd-5vfjq" Oct 30 07:35:11.717955 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:11.717926 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95jlm\" (UniqueName: \"kubernetes.io/projected/33977737-ec70-47e5-a689-4c471c77c42e-kube-api-access-95jlm\") pod \"payload-processing-e2e-extmod-a-1e0880-79484bb6bd-5vfjq\" (UID: \"33977737-ec70-47e5-a689-4c471c77c42e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1e0880-79484bb6bd-5vfjq" Oct 30 07:35:11.853360 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:11.853326 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1e0880-79484bb6bd-5vfjq" Oct 30 07:35:11.879298 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:35:11.879270 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ba04d1_bbfe_4ed3_b0b2_55d554a025fb.slice/crio-9b626ee5fdbd346ba4d3b2520dd3234309cd4c0a8f15ec86ec71a210055b673b WatchSource:0}: Error finding container 9b626ee5fdbd346ba4d3b2520dd3234309cd4c0a8f15ec86ec71a210055b673b: Status 404 returned error can't find the container with id 9b626ee5fdbd346ba4d3b2520dd3234309cd4c0a8f15ec86ec71a210055b673b Oct 30 07:35:11.881482 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:11.881448 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1e0880-6946fccf5f-2rcnf"] Oct 30 07:35:11.994281 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:11.994236 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1e0880-6946fccf5f-2rcnf" event={"ID":"64ba04d1-bbfe-4ed3-b0b2-55d554a025fb","Type":"ContainerStarted","Data":"c7c773054675033d5f0b1cbd172417a233d38b94b95b6560c349a423282a997b"} Oct 30 07:35:11.994469 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:11.994312 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1e0880-6946fccf5f-2rcnf" event={"ID":"64ba04d1-bbfe-4ed3-b0b2-55d554a025fb","Type":"ContainerStarted","Data":"9b626ee5fdbd346ba4d3b2520dd3234309cd4c0a8f15ec86ec71a210055b673b"} Oct 30 07:35:11.995554 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:11.995526 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1e0880-6946fccf5f-2rcnf" Oct 30 07:35:12.041729 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:12.041660 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1e0880-6946fccf5f-2rcnf" podStartSLOduration=1.041638752 podStartE2EDuration="1.041638752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:35:12.040431383 +0000 UTC m=+2318.383479374" watchObservedRunningTime="2026-10-30 07:35:12.041638752 +0000 UTC m=+2318.384686717" Oct 30 07:35:12.211830 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:12.210561 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1e0880-79484bb6bd-5vfjq"] Oct 30 07:35:13.005012 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:13.004971 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1e0880-79484bb6bd-5vfjq" event={"ID":"33977737-ec70-47e5-a689-4c471c77c42e","Type":"ContainerStarted","Data":"e0cd5e62032638d8f0203a10f7dccac3d09d50a29ee671b3f5abc055bab6b1d9"} Oct 30 07:35:13.005237 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:13.005042 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1e0880-79484bb6bd-5vfjq" event={"ID":"33977737-ec70-47e5-a689-4c471c77c42e","Type":"ContainerStarted","Data":"74ffc1b362708be106a837a5e5a89be0e21d575a0d5510c1c611c5806df9a490"} Oct 30 07:35:13.005237 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:13.005073 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-1e0880-79484bb6bd-5vfjq" Oct 30 07:35:13.051067 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:13.050992 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-1e0880-79484bb6bd-5vfjq" podStartSLOduration=2.05097083 podStartE2EDuration="2.05097083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:35:13.049276069 +0000 UTC m=+2319.392324036" watchObservedRunningTime="2026-10-30 07:35:13.05097083 +0000 UTC m=+2319.394018796" Oct 30 07:35:17.180633 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:17.180599 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7"] Oct 30 07:35:17.181030 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:17.180810 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" podUID="9ba10610-90eb-43ad-b937-e6c4d4a9dfb5" containerName="istio-proxy" containerID="cri-o://253c9c2ae58366494085b687f9c461607d701ff9ed9d6cf8b0cce3df4f51ea87" gracePeriod=30 Oct 30 07:35:22.506957 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:22.506925 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:35:22.693157 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:22.693117 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-kube-api-access-76nhv\" (UniqueName: \"kubernetes.io/projected/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-kube-api-access-76nhv\") pod \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " Oct 30 07:35:22.693351 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:22.693186 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istio-podinfo\") pod \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " Oct 30 07:35:22.693351 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:22.693217 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istio-token\" (UniqueName: \"kubernetes.io/projected/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istio-token\") pod \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " Oct 30 07:35:22.693351 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:22.693293 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-credential-socket\") pod \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " Oct 30 07:35:22.693543 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:22.693359 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istio-data\") pod \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " Oct 30 07:35:22.693543 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:22.693387 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istio-envoy\") pod \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " Oct 30 07:35:22.693688 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:22.693599 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-workload-socket\") pod \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " Oct 30 07:35:22.693688 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:22.693644 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-workload-certs\") pod \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " Oct 30 07:35:22.693688 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:22.693667 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istiod-ca-cert\") pod \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\" (UID: \"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5\") " Oct 30 07:35:22.693879 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:22.693832 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istio-data" pod "9ba10610-90eb-43ad-b937-e6c4d4a9dfb5" (UID: "9ba10610-90eb-43ad-b937-e6c4d4a9dfb5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:22.694290 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:22.694265 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:35:22.694429 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:22.694364 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-credential-socket" pod "9ba10610-90eb-43ad-b937-e6c4d4a9dfb5" (UID: "9ba10610-90eb-43ad-b937-e6c4d4a9dfb5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:22.694645 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:22.694543 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-workload-certs" pod "9ba10610-90eb-43ad-b937-e6c4d4a9dfb5" (UID: "9ba10610-90eb-43ad-b937-e6c4d4a9dfb5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:22.694762 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:22.694671 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-workload-socket" pod "9ba10610-90eb-43ad-b937-e6c4d4a9dfb5" (UID: "9ba10610-90eb-43ad-b937-e6c4d4a9dfb5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:22.694762 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:22.694701 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istiod-ca-cert" pod "9ba10610-90eb-43ad-b937-e6c4d4a9dfb5" (UID: "9ba10610-90eb-43ad-b937-e6c4d4a9dfb5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:35:22.696832 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:22.696802 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istio-token" pod "9ba10610-90eb-43ad-b937-e6c4d4a9dfb5" (UID: "9ba10610-90eb-43ad-b937-e6c4d4a9dfb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:35:22.696986 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:22.696966 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-kube-api-access-76nhv" pod "9ba10610-90eb-43ad-b937-e6c4d4a9dfb5" (UID: "9ba10610-90eb-43ad-b937-e6c4d4a9dfb5"). InnerVolumeSpecName "kube-api-access-76nhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:35:22.698239 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:22.698213 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istio-podinfo" pod "9ba10610-90eb-43ad-b937-e6c4d4a9dfb5" (UID: "9ba10610-90eb-43ad-b937-e6c4d4a9dfb5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:35:22.698773 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:22.698739 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istio-envoy" pod "9ba10610-90eb-43ad-b937-e6c4d4a9dfb5" (UID: "9ba10610-90eb-43ad-b937-e6c4d4a9dfb5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:22.796061 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:22.796016 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76nhv\" (UniqueName: \"kubernetes.io/projected/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-kube-api-access-76nhv\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:35:22.796061 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:22.796056 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:35:22.796061 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:22.796069 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:35:22.796327 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:22.796085 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:35:22.796327 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:22.796107 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:35:22.796327 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:22.796120 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:35:22.796327 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:22.796132 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:35:22.796327 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:22.796153 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:35:22.967623 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:22.967549 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt"] Oct 30 07:35:22.968672 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:22.968623 4951 cpu_manager.go:414] "containerMap: removing container" podUID="9ba10610-90eb-43ad-b937-e6c4d4a9dfb5" containerName="istio-proxy" Oct 30 07:35:22.968672 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:22.968649 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ba10610-90eb-43ad-b937-e6c4d4a9dfb5" containerName="istio-proxy" Oct 30 07:35:22.968936 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:22.968855 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ba10610-90eb-43ad-b937-e6c4d4a9dfb5" containerName="istio-proxy" Oct 30 07:35:22.974653 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:22.974628 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:35:23.002894 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.002431 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt"] Oct 30 07:35:23.034539 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.034508 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-228571-openshift-default-dockercfg-ntx22\"" Oct 30 07:35:23.071740 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:35:23.071657 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:35:23.100980 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.100940 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-workload-certs\") pod \"e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:35:23.101202 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.100993 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-workload-socket\") pod \"e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:35:23.101202 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.101188 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istio-data\") pod \"e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:35:23.101333 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.101229 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istio-token\") pod \"e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:35:23.101470 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.101440 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-credential-socket\") pod \"e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:35:23.101628 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.101486 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istio-envoy\") pod \"e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:35:23.101628 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.101561 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istio-podinfo\") pod \"e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:35:23.101765 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.101698 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcpw9\" (UniqueName: \"kubernetes.io/projected/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-kube-api-access-hcpw9\") pod \"e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:35:23.101818 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.101768 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istiod-ca-cert\") pod \"e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:35:23.104905 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.104881 4951 generic.go:356] "Generic (PLEG): container finished" podID="9ba10610-90eb-43ad-b937-e6c4d4a9dfb5" containerID="253c9c2ae58366494085b687f9c461607d701ff9ed9d6cf8b0cce3df4f51ea87" exitCode=0 Oct 30 07:35:23.105057 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.104947 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" Oct 30 07:35:23.105057 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.104945 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" event={"ID":"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5","Type":"ContainerDied","Data":"253c9c2ae58366494085b687f9c461607d701ff9ed9d6cf8b0cce3df4f51ea87"} Oct 30 07:35:23.105169 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.105076 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7" event={"ID":"9ba10610-90eb-43ad-b937-e6c4d4a9dfb5","Type":"ContainerDied","Data":"20c78162cb20f4c5a5414e2ac04cbeb94d3727826342e7037e1b6e02e05ff623"} Oct 30 07:35:23.105169 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.105108 4951 scope.go:122] "RemoveContainer" containerID="253c9c2ae58366494085b687f9c461607d701ff9ed9d6cf8b0cce3df4f51ea87" Oct 30 07:35:23.117758 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.117731 4951 scope.go:122] "RemoveContainer" containerID="253c9c2ae58366494085b687f9c461607d701ff9ed9d6cf8b0cce3df4f51ea87" Oct 30 07:35:23.118126 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:35:23.118101 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"253c9c2ae58366494085b687f9c461607d701ff9ed9d6cf8b0cce3df4f51ea87\": container with ID starting with 253c9c2ae58366494085b687f9c461607d701ff9ed9d6cf8b0cce3df4f51ea87 not found: ID does not exist" containerID="253c9c2ae58366494085b687f9c461607d701ff9ed9d6cf8b0cce3df4f51ea87" Oct 30 07:35:23.118214 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.118136 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"253c9c2ae58366494085b687f9c461607d701ff9ed9d6cf8b0cce3df4f51ea87"} err="failed to get container status \"253c9c2ae58366494085b687f9c461607d701ff9ed9d6cf8b0cce3df4f51ea87\": rpc error: code = NotFound desc = could not find container \"253c9c2ae58366494085b687f9c461607d701ff9ed9d6cf8b0cce3df4f51ea87\": container with ID starting with 253c9c2ae58366494085b687f9c461607d701ff9ed9d6cf8b0cce3df4f51ea87 not found: ID does not exist" Oct 30 07:35:23.232597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.228050 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istiod-ca-cert\") pod \"e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:35:23.232597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.228131 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-workload-certs\") pod \"e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:35:23.232597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.228163 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-workload-socket\") pod \"e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:35:23.232597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.228237 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istio-data\") pod \"e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:35:23.232597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.228266 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istio-token\") pod \"e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:35:23.232597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.228411 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-credential-socket\") pod \"e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:35:23.232597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.228437 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istio-envoy\") pod \"e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:35:23.232597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.228500 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istio-podinfo\") pod \"e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:35:23.232597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.228565 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcpw9\" (UniqueName: \"kubernetes.io/projected/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-kube-api-access-hcpw9\") pod \"e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:35:23.232597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.229936 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istiod-ca-cert\") pod \"e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:35:23.232597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.230236 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-workload-certs\") pod \"e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:35:23.232597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.230460 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-workload-socket\") pod \"e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:35:23.254838 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.254796 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7"] Oct 30 07:35:23.319597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.311609 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-wb8n7"] Oct 30 07:35:23.319597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.276476 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istio-data\") pod \"e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:35:23.319597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.267011 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-credential-socket\") pod \"e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:35:23.319597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.315689 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcpw9\" (UniqueName: \"kubernetes.io/projected/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-kube-api-access-hcpw9\") pod \"e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:35:23.342656 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.332012 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istio-envoy\") pod \"e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:35:23.371974 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.359670 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istio-podinfo\") pod \"e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:35:23.371974 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.368159 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istio-token\") pod \"e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:35:23.601803 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.601717 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:35:23.774005 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.773962 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ba10610-90eb-43ad-b937-e6c4d4a9dfb5" path="/var/lib/kubelet/pods/9ba10610-90eb-43ad-b937-e6c4d4a9dfb5/volumes" Oct 30 07:35:23.968979 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.968321 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt"] Oct 30 07:35:23.970785 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:35:23.970748 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf4c56f9_5d9c_40a0_8cd4_490d8459e4c4.slice/crio-9dfc129ab83cfe45f21f5709f2e75324802b3fdad4020e112291f2473c5072cf WatchSource:0}: Error finding container 9dfc129ab83cfe45f21f5709f2e75324802b3fdad4020e112291f2473c5072cf: Status 404 returned error can't find the container with id 9dfc129ab83cfe45f21f5709f2e75324802b3fdad4020e112291f2473c5072cf Oct 30 07:35:23.973912 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.973870 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:35:23.974413 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.974377 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:35:23.974529 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:23.974439 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:35:24.017703 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:24.017674 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1e0880-6946fccf5f-2rcnf" Oct 30 07:35:24.017870 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:24.017760 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-1e0880-79484bb6bd-5vfjq" Oct 30 07:35:24.153527 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:24.153488 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" event={"ID":"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4","Type":"ContainerStarted","Data":"91a57d17aefb7d1502c0f545ac473c165cd5beada4caf3c52a66d044476b5836"} Oct 30 07:35:24.153804 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:24.153778 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" event={"ID":"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4","Type":"ContainerStarted","Data":"9dfc129ab83cfe45f21f5709f2e75324802b3fdad4020e112291f2473c5072cf"} Oct 30 07:35:24.205870 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:24.205792 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" podStartSLOduration=2.205772839 podStartE2EDuration="2.205772839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:35:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:35:24.204726347 +0000 UTC m=+2330.547774312" watchObservedRunningTime="2026-10-30 07:35:24.205772839 +0000 UTC m=+2330.548820806" Oct 30 07:35:24.601901 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:24.601869 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:35:25.435016 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:25.433005 4951 patch_prober.go:29] interesting pod/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": read tcp 10.132.0.2:35996->10.132.0.208:15021: read: connection reset by peer" start-of-body= Oct 30 07:35:25.435016 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:25.433075 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" podUID="8562e54e-5cb6-4805-ae61-afc7b1d5c0c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": read tcp 10.132.0.2:35996->10.132.0.208:15021: read: connection reset by peer" Oct 30 07:35:25.609466 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:25.609379 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:35:26.172254 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:26.172219 4951 generic.go:356] "Generic (PLEG): container finished" podID="8562e54e-5cb6-4805-ae61-afc7b1d5c0c6" containerID="298585ce83863ba43af616efbe835894bca8b711581bdc06ca7bff1d3ab975b7" exitCode=0 Oct 30 07:35:26.172459 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:26.172288 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" event={"ID":"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6","Type":"ContainerDied","Data":"298585ce83863ba43af616efbe835894bca8b711581bdc06ca7bff1d3ab975b7"} Oct 30 07:35:26.172504 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:26.172479 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:35:26.172806 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:26.172791 4951 scope.go:122] "RemoveContainer" containerID="298585ce83863ba43af616efbe835894bca8b711581bdc06ca7bff1d3ab975b7" Oct 30 07:35:26.173796 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:26.173408 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:35:26.174402 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:26.174375 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:35:26.174466 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:26.174446 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:35:26.174507 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:26.174480 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:35:27.404657 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:27.402658 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" event={"ID":"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6","Type":"ContainerStarted","Data":"32df6e73728e70dbf4bb3d0c6008a7183be84aac4919ea08498ad029d6f4c7d8"} Oct 30 07:35:28.183751 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:28.183709 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:35:28.183974 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:28.183927 4951 patch_prober.go:29] interesting pod/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 07:35:28.184119 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:28.183970 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" podUID="8562e54e-5cb6-4805-ae61-afc7b1d5c0c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 07:35:29.184174 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:29.184137 4951 patch_prober.go:29] interesting pod/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 07:35:29.184611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:29.184194 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" podUID="8562e54e-5cb6-4805-ae61-afc7b1d5c0c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 07:35:29.660243 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:29.660197 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-228571-676b499bc-k76j6"] Oct 30 07:35:29.670725 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:29.670689 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-228571-676b499bc-k76j6" Oct 30 07:35:29.688455 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:29.688422 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-228571-676b499bc-k76j6"] Oct 30 07:35:29.697324 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:29.697296 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-228571\"" Oct 30 07:35:29.698679 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:29.698648 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-228571-dockercfg-mjb5c\"" Oct 30 07:35:29.708809 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:29.708724 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c290e9a-af71-48e8-99c6-43056987a08d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-228571-676b499bc-k76j6\" (UID: \"9c290e9a-af71-48e8-99c6-43056987a08d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-228571-676b499bc-k76j6" Oct 30 07:35:29.709202 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:29.709003 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr5wx\" (UniqueName: \"kubernetes.io/projected/9c290e9a-af71-48e8-99c6-43056987a08d-kube-api-access-cr5wx\") pod \"payload-pre-processing-e2e-shared-b-w0-228571-676b499bc-k76j6\" (UID: \"9c290e9a-af71-48e8-99c6-43056987a08d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-228571-676b499bc-k76j6" Oct 30 07:35:29.863600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:29.824404 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr5wx\" (UniqueName: \"kubernetes.io/projected/9c290e9a-af71-48e8-99c6-43056987a08d-kube-api-access-cr5wx\") pod \"payload-pre-processing-e2e-shared-b-w0-228571-676b499bc-k76j6\" (UID: \"9c290e9a-af71-48e8-99c6-43056987a08d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-228571-676b499bc-k76j6" Oct 30 07:35:29.881605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:29.864126 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c290e9a-af71-48e8-99c6-43056987a08d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-228571-676b499bc-k76j6\" (UID: \"9c290e9a-af71-48e8-99c6-43056987a08d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-228571-676b499bc-k76j6" Oct 30 07:35:29.907592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:29.874480 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c290e9a-af71-48e8-99c6-43056987a08d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-228571-676b499bc-k76j6\" (UID: \"9c290e9a-af71-48e8-99c6-43056987a08d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-228571-676b499bc-k76j6" Oct 30 07:35:29.964696 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:29.917782 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-228571-5995bc74cb-5kvfx"] Oct 30 07:35:29.977686 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:29.977655 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-228571-5995bc74cb-5kvfx" Oct 30 07:35:30.013608 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:29.991830 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr5wx\" (UniqueName: \"kubernetes.io/projected/9c290e9a-af71-48e8-99c6-43056987a08d-kube-api-access-cr5wx\") pod \"payload-pre-processing-e2e-shared-b-w0-228571-676b499bc-k76j6\" (UID: \"9c290e9a-af71-48e8-99c6-43056987a08d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-228571-676b499bc-k76j6" Oct 30 07:35:30.013608 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:29.993592 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-228571-676b499bc-k76j6" Oct 30 07:35:30.085205 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:30.085179 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-228571-5995bc74cb-5kvfx"] Oct 30 07:35:30.095112 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:30.095044 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/399020ed-d53c-41b2-8b3d-298647396c87-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-228571-5995bc74cb-5kvfx\" (UID: \"399020ed-d53c-41b2-8b3d-298647396c87\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-228571-5995bc74cb-5kvfx" Oct 30 07:35:30.095501 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:30.095465 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrqgk\" (UniqueName: \"kubernetes.io/projected/399020ed-d53c-41b2-8b3d-298647396c87-kube-api-access-qrqgk\") pod \"payload-processing-e2e-shared-b-w0-228571-5995bc74cb-5kvfx\" (UID: \"399020ed-d53c-41b2-8b3d-298647396c87\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-228571-5995bc74cb-5kvfx" Oct 30 07:35:30.187193 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:30.184720 4951 patch_prober.go:29] interesting pod/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 07:35:30.187193 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:30.184784 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" podUID="8562e54e-5cb6-4805-ae61-afc7b1d5c0c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 07:35:30.197159 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:30.197119 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/399020ed-d53c-41b2-8b3d-298647396c87-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-228571-5995bc74cb-5kvfx\" (UID: \"399020ed-d53c-41b2-8b3d-298647396c87\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-228571-5995bc74cb-5kvfx" Oct 30 07:35:30.197348 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:30.197225 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrqgk\" (UniqueName: \"kubernetes.io/projected/399020ed-d53c-41b2-8b3d-298647396c87-kube-api-access-qrqgk\") pod \"payload-processing-e2e-shared-b-w0-228571-5995bc74cb-5kvfx\" (UID: \"399020ed-d53c-41b2-8b3d-298647396c87\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-228571-5995bc74cb-5kvfx" Oct 30 07:35:30.198914 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:30.198886 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/399020ed-d53c-41b2-8b3d-298647396c87-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-228571-5995bc74cb-5kvfx\" (UID: \"399020ed-d53c-41b2-8b3d-298647396c87\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-228571-5995bc74cb-5kvfx" Oct 30 07:35:30.259673 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:30.254903 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrqgk\" (UniqueName: \"kubernetes.io/projected/399020ed-d53c-41b2-8b3d-298647396c87-kube-api-access-qrqgk\") pod \"payload-processing-e2e-shared-b-w0-228571-5995bc74cb-5kvfx\" (UID: \"399020ed-d53c-41b2-8b3d-298647396c87\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-228571-5995bc74cb-5kvfx" Oct 30 07:35:30.397677 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:30.397644 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-228571-5995bc74cb-5kvfx" Oct 30 07:35:30.458888 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:35:30.447768 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c290e9a_af71_48e8_99c6_43056987a08d.slice/crio-cab3d0b6a2e1f5a23715a19f048c1ad303435c95caee94dae757709cd1771975 WatchSource:0}: Error finding container cab3d0b6a2e1f5a23715a19f048c1ad303435c95caee94dae757709cd1771975: Status 404 returned error can't find the container with id cab3d0b6a2e1f5a23715a19f048c1ad303435c95caee94dae757709cd1771975 Oct 30 07:35:30.458888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:30.456587 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-228571-676b499bc-k76j6"] Oct 30 07:35:30.757072 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:35:30.756493 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod399020ed_d53c_41b2_8b3d_298647396c87.slice/crio-dc54c976b1438b3ab3c70c9f9bbeeac57b5608e8e3120546b736c953e7ae1481 WatchSource:0}: Error finding container dc54c976b1438b3ab3c70c9f9bbeeac57b5608e8e3120546b736c953e7ae1481: Status 404 returned error can't find the container with id dc54c976b1438b3ab3c70c9f9bbeeac57b5608e8e3120546b736c953e7ae1481 Oct 30 07:35:30.760556 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:30.760529 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-228571-5995bc74cb-5kvfx"] Oct 30 07:35:31.184079 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:31.183992 4951 patch_prober.go:29] interesting pod/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 07:35:31.184079 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:31.184042 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" podUID="8562e54e-5cb6-4805-ae61-afc7b1d5c0c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 07:35:31.276180 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:31.276142 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-228571-5995bc74cb-5kvfx" event={"ID":"399020ed-d53c-41b2-8b3d-298647396c87","Type":"ContainerStarted","Data":"763035ff78ae4ceeec5f0ec9862891754f440b7ea2ce90a6e4e9044b405f7131"} Oct 30 07:35:31.276727 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:31.276236 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-228571-5995bc74cb-5kvfx" event={"ID":"399020ed-d53c-41b2-8b3d-298647396c87","Type":"ContainerStarted","Data":"dc54c976b1438b3ab3c70c9f9bbeeac57b5608e8e3120546b736c953e7ae1481"} Oct 30 07:35:31.276727 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:31.276262 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-228571-5995bc74cb-5kvfx" Oct 30 07:35:31.278111 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:31.278085 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-228571-676b499bc-k76j6" event={"ID":"9c290e9a-af71-48e8-99c6-43056987a08d","Type":"ContainerStarted","Data":"11972b0894266cf0bda6b40d698fcaa63a51bfc34ae04ff127fdc891b8fa6faf"} Oct 30 07:35:31.278255 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:31.278126 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-228571-676b499bc-k76j6" event={"ID":"9c290e9a-af71-48e8-99c6-43056987a08d","Type":"ContainerStarted","Data":"cab3d0b6a2e1f5a23715a19f048c1ad303435c95caee94dae757709cd1771975"} Oct 30 07:35:31.278459 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:31.278441 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-228571-676b499bc-k76j6" Oct 30 07:35:31.331598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:31.321837 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-228571-5995bc74cb-5kvfx" podStartSLOduration=2.321819356 podStartE2EDuration="2.321819356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:35:31.320658568 +0000 UTC m=+2337.663706533" watchObservedRunningTime="2026-10-30 07:35:31.321819356 +0000 UTC m=+2337.664867322" Oct 30 07:35:31.379819 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:31.379761 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-228571-676b499bc-k76j6" podStartSLOduration=2.379745787 podStartE2EDuration="2.379745787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:35:31.377764593 +0000 UTC m=+2337.720812558" watchObservedRunningTime="2026-10-30 07:35:31.379745787 +0000 UTC m=+2337.722793752" Oct 30 07:35:32.184605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:32.184552 4951 patch_prober.go:29] interesting pod/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 07:35:32.184801 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:32.184629 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" podUID="8562e54e-5cb6-4805-ae61-afc7b1d5c0c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 07:35:33.189068 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:33.189030 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:35:33.189479 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:33.189457 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:35:33.190174 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:33.190154 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 07:35:42.312647 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:42.312324 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-228571-676b499bc-k76j6" Oct 30 07:35:42.313186 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:42.312711 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-228571-5995bc74cb-5kvfx" Oct 30 07:35:47.401141 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:47.401104 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8"] Oct 30 07:35:47.407840 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:47.407817 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:35:47.438551 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:47.438482 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-b096ec-openshift-default-dockercfg-hgdlw\"" Oct 30 07:35:47.456606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:47.456524 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8"] Oct 30 07:35:47.515347 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:47.515300 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-credential-socket\") pod \"e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:35:47.515549 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:47.515379 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxpzg\" (UniqueName: \"kubernetes.io/projected/3ad4d41c-7af3-453d-9005-d34264ed3acd-kube-api-access-nxpzg\") pod \"e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:35:47.515549 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:47.515528 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad4d41c-7af3-453d-9005-d34264ed3acd-istio-token\") pod \"e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:35:47.515788 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:47.515643 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad4d41c-7af3-453d-9005-d34264ed3acd-istiod-ca-cert\") pod \"e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:35:47.515788 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:47.515681 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-istio-data\") pod \"e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:35:47.515788 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:47.515712 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad4d41c-7af3-453d-9005-d34264ed3acd-istio-podinfo\") pod \"e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:35:47.515788 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:47.515755 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-istio-envoy\") pod \"e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:35:47.516044 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:47.515785 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-workload-certs\") pod \"e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:35:47.516044 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:47.515875 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-workload-socket\") pod \"e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:35:47.623595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:47.622872 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad4d41c-7af3-453d-9005-d34264ed3acd-istio-token\") pod \"e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:35:47.623595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:47.623002 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad4d41c-7af3-453d-9005-d34264ed3acd-istiod-ca-cert\") pod \"e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:35:47.623595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:47.623048 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-istio-data\") pod \"e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:35:47.623595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:47.623078 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad4d41c-7af3-453d-9005-d34264ed3acd-istio-podinfo\") pod \"e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:35:47.623595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:47.623127 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-istio-envoy\") pod \"e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:35:47.623595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:47.623153 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-workload-certs\") pod \"e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:35:47.623595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:47.623243 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-workload-socket\") pod \"e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:35:47.623595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:47.623296 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-credential-socket\") pod \"e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:35:47.623595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:47.623340 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxpzg\" (UniqueName: \"kubernetes.io/projected/3ad4d41c-7af3-453d-9005-d34264ed3acd-kube-api-access-nxpzg\") pod \"e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:35:47.634387 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:47.627872 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-workload-socket\") pod \"e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:35:47.634387 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:47.628133 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-workload-certs\") pod \"e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:35:47.634387 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:47.628396 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-istio-data\") pod \"e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:35:47.634387 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:47.628673 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-credential-socket\") pod \"e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:35:47.638966 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:47.638926 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-istio-envoy\") pod \"e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:35:47.648448 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:47.646806 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad4d41c-7af3-453d-9005-d34264ed3acd-istio-podinfo\") pod \"e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:35:47.657639 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:47.649407 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad4d41c-7af3-453d-9005-d34264ed3acd-istiod-ca-cert\") pod \"e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:35:47.663804 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:47.663743 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad4d41c-7af3-453d-9005-d34264ed3acd-istio-token\") pod \"e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:35:47.680081 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:47.678299 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxpzg\" (UniqueName: \"kubernetes.io/projected/3ad4d41c-7af3-453d-9005-d34264ed3acd-kube-api-access-nxpzg\") pod \"e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:35:47.742594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:47.728622 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:35:48.100545 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:35:48.100516 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ad4d41c_7af3_453d_9005_d34264ed3acd.slice/crio-86a9584aa9c2aca2c746bdc80326d7290b4aab6c5a224c186638a03c383249de WatchSource:0}: Error finding container 86a9584aa9c2aca2c746bdc80326d7290b4aab6c5a224c186638a03c383249de: Status 404 returned error can't find the container with id 86a9584aa9c2aca2c746bdc80326d7290b4aab6c5a224c186638a03c383249de Oct 30 07:35:48.101301 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:48.101276 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8"] Oct 30 07:35:48.103323 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:48.103292 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:35:48.103380 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:48.103356 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:35:48.103435 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:48.103389 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:35:48.456052 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:48.455959 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" event={"ID":"3ad4d41c-7af3-453d-9005-d34264ed3acd","Type":"ContainerStarted","Data":"818324153ca59fed54d04cb66f35f25f4dcb53b96e013b9b2fa0690353ed7c60"} Oct 30 07:35:48.456052 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:48.456028 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" event={"ID":"3ad4d41c-7af3-453d-9005-d34264ed3acd","Type":"ContainerStarted","Data":"86a9584aa9c2aca2c746bdc80326d7290b4aab6c5a224c186638a03c383249de"} Oct 30 07:35:48.509868 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:48.509811 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" podStartSLOduration=1.509797082 podStartE2EDuration="1.509797082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:35:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:35:48.504239493 +0000 UTC m=+2354.847287484" watchObservedRunningTime="2026-10-30 07:35:48.509797082 +0000 UTC m=+2354.852845046" Oct 30 07:35:48.730411 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:48.730336 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:35:49.736682 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:49.736642 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:35:50.471014 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:50.470989 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:35:50.471910 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:50.471877 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:35:53.148753 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:35:53.148716 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:35:54.420164 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:54.420129 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd"] Oct 30 07:35:54.424800 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:54.424778 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd" Oct 30 07:35:54.446629 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:54.445155 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-228571-serving-cert\"" Oct 30 07:35:54.465060 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:54.465015 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd"] Oct 30 07:35:54.504861 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:54.504818 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vx8t\" (UniqueName: \"kubernetes.io/projected/0628b544-4ce0-4902-a5e8-5fef9194d1a2-kube-api-access-4vx8t\") pod \"maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd\" (UID: \"0628b544-4ce0-4902-a5e8-5fef9194d1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd" Oct 30 07:35:54.505312 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:54.505282 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0628b544-4ce0-4902-a5e8-5fef9194d1a2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd\" (UID: \"0628b544-4ce0-4902-a5e8-5fef9194d1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd" Oct 30 07:35:54.505527 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:54.505497 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0628b544-4ce0-4902-a5e8-5fef9194d1a2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd\" (UID: \"0628b544-4ce0-4902-a5e8-5fef9194d1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd" Oct 30 07:35:54.505688 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:54.505626 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0628b544-4ce0-4902-a5e8-5fef9194d1a2-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd\" (UID: \"0628b544-4ce0-4902-a5e8-5fef9194d1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd" Oct 30 07:35:54.606743 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:54.606710 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0628b544-4ce0-4902-a5e8-5fef9194d1a2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd\" (UID: \"0628b544-4ce0-4902-a5e8-5fef9194d1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd" Oct 30 07:35:54.606743 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:54.606759 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0628b544-4ce0-4902-a5e8-5fef9194d1a2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd\" (UID: \"0628b544-4ce0-4902-a5e8-5fef9194d1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd" Oct 30 07:35:54.607017 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:54.606813 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0628b544-4ce0-4902-a5e8-5fef9194d1a2-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd\" (UID: \"0628b544-4ce0-4902-a5e8-5fef9194d1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd" Oct 30 07:35:54.607017 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:54.606982 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vx8t\" (UniqueName: \"kubernetes.io/projected/0628b544-4ce0-4902-a5e8-5fef9194d1a2-kube-api-access-4vx8t\") pod \"maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd\" (UID: \"0628b544-4ce0-4902-a5e8-5fef9194d1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd" Oct 30 07:35:54.607505 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:54.607487 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0628b544-4ce0-4902-a5e8-5fef9194d1a2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd\" (UID: \"0628b544-4ce0-4902-a5e8-5fef9194d1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd" Oct 30 07:35:54.609265 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:54.609241 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0628b544-4ce0-4902-a5e8-5fef9194d1a2-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd\" (UID: \"0628b544-4ce0-4902-a5e8-5fef9194d1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd" Oct 30 07:35:54.609394 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:54.609291 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0628b544-4ce0-4902-a5e8-5fef9194d1a2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd\" (UID: \"0628b544-4ce0-4902-a5e8-5fef9194d1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd" Oct 30 07:35:54.660465 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:54.660397 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vx8t\" (UniqueName: \"kubernetes.io/projected/0628b544-4ce0-4902-a5e8-5fef9194d1a2-kube-api-access-4vx8t\") pod \"maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd\" (UID: \"0628b544-4ce0-4902-a5e8-5fef9194d1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd" Oct 30 07:35:54.741513 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:54.741174 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd" Oct 30 07:35:55.052990 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:35:55.052957 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0628b544_4ce0_4902_a5e8_5fef9194d1a2.slice/crio-48d5bb2313c13c9dd4c8581bbf8ad51743d69d786827cb7ca86d2eb707def89b WatchSource:0}: Error finding container 48d5bb2313c13c9dd4c8581bbf8ad51743d69d786827cb7ca86d2eb707def89b: Status 404 returned error can't find the container with id 48d5bb2313c13c9dd4c8581bbf8ad51743d69d786827cb7ca86d2eb707def89b Oct 30 07:35:55.053470 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:55.053447 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd"] Oct 30 07:35:55.506709 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:55.506675 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd" event={"ID":"0628b544-4ce0-4902-a5e8-5fef9194d1a2","Type":"ContainerStarted","Data":"1345c4ee041facb5882df14d6b7bd55e683ae3e959ae8ee66f184b8daaf4a953"} Oct 30 07:35:55.507208 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:55.506726 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd" event={"ID":"0628b544-4ce0-4902-a5e8-5fef9194d1a2","Type":"ContainerStarted","Data":"48d5bb2313c13c9dd4c8581bbf8ad51743d69d786827cb7ca86d2eb707def89b"} Oct 30 07:35:55.507208 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:55.506769 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd" Oct 30 07:35:55.575599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:35:55.571672 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd" podStartSLOduration=1.23940914 podStartE2EDuration="1.571647059s" totalImagesPullingTime="332.237919ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:35:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:35:55.552008307 +0000 UTC m=+2361.895056294" watchObservedRunningTime="2026-10-30 07:35:55.571647059 +0000 UTC m=+2361.914695026" Oct 30 07:36:00.935834 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:00.935796 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82"] Oct 30 07:36:00.936412 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:00.936045 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" podUID="9328c8af-90ee-4949-be97-f3e6c04416a2" containerName="istio-proxy" containerID="cri-o://ca0668842600a7ce99755aa25e95e315540eb32a426b75ab9aabfc795bee82a5" gracePeriod=30 Oct 30 07:36:01.551594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:01.551244 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd" Oct 30 07:36:06.233137 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.233046 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:36:06.335221 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.335182 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-istio-envoy\") pod \"9328c8af-90ee-4949-be97-f3e6c04416a2\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " Oct 30 07:36:06.335421 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.335313 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-istio-data\") pod \"9328c8af-90ee-4949-be97-f3e6c04416a2\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " Oct 30 07:36:06.335421 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.335380 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9328c8af-90ee-4949-be97-f3e6c04416a2-kube-api-access-mxcr6\" (UniqueName: \"kubernetes.io/projected/9328c8af-90ee-4949-be97-f3e6c04416a2-kube-api-access-mxcr6\") pod \"9328c8af-90ee-4949-be97-f3e6c04416a2\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " Oct 30 07:36:06.335552 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.335456 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9328c8af-90ee-4949-be97-f3e6c04416a2-istio-token\" (UniqueName: \"kubernetes.io/projected/9328c8af-90ee-4949-be97-f3e6c04416a2-istio-token\") pod \"9328c8af-90ee-4949-be97-f3e6c04416a2\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " Oct 30 07:36:06.335629 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.335549 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9328c8af-90ee-4949-be97-f3e6c04416a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9328c8af-90ee-4949-be97-f3e6c04416a2-istiod-ca-cert\") pod \"9328c8af-90ee-4949-be97-f3e6c04416a2\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " Oct 30 07:36:06.335629 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.335617 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-istio-data" pod "9328c8af-90ee-4949-be97-f3e6c04416a2" (UID: "9328c8af-90ee-4949-be97-f3e6c04416a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:06.335772 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.335629 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-credential-socket\") pod \"9328c8af-90ee-4949-be97-f3e6c04416a2\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " Oct 30 07:36:06.335772 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.335760 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-workload-certs\") pod \"9328c8af-90ee-4949-be97-f3e6c04416a2\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " Oct 30 07:36:06.335859 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.335844 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9328c8af-90ee-4949-be97-f3e6c04416a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9328c8af-90ee-4949-be97-f3e6c04416a2-istio-podinfo\") pod \"9328c8af-90ee-4949-be97-f3e6c04416a2\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " Oct 30 07:36:06.335908 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.335890 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-workload-socket\") pod \"9328c8af-90ee-4949-be97-f3e6c04416a2\" (UID: \"9328c8af-90ee-4949-be97-f3e6c04416a2\") " Oct 30 07:36:06.336532 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.336511 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:06.337237 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.337189 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9328c8af-90ee-4949-be97-f3e6c04416a2-istiod-ca-cert" pod "9328c8af-90ee-4949-be97-f3e6c04416a2" (UID: "9328c8af-90ee-4949-be97-f3e6c04416a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:36:06.337708 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.337678 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-workload-certs" pod "9328c8af-90ee-4949-be97-f3e6c04416a2" (UID: "9328c8af-90ee-4949-be97-f3e6c04416a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:06.337821 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.337716 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-workload-socket" pod "9328c8af-90ee-4949-be97-f3e6c04416a2" (UID: "9328c8af-90ee-4949-be97-f3e6c04416a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:06.338548 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.338522 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-credential-socket" pod "9328c8af-90ee-4949-be97-f3e6c04416a2" (UID: "9328c8af-90ee-4949-be97-f3e6c04416a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:06.339130 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.339104 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9328c8af-90ee-4949-be97-f3e6c04416a2-istio-token" pod "9328c8af-90ee-4949-be97-f3e6c04416a2" (UID: "9328c8af-90ee-4949-be97-f3e6c04416a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:36:06.339372 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.339353 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-istio-envoy" pod "9328c8af-90ee-4949-be97-f3e6c04416a2" (UID: "9328c8af-90ee-4949-be97-f3e6c04416a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:06.340152 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.340128 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9328c8af-90ee-4949-be97-f3e6c04416a2-istio-podinfo" pod "9328c8af-90ee-4949-be97-f3e6c04416a2" (UID: "9328c8af-90ee-4949-be97-f3e6c04416a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:36:06.342745 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.342715 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9328c8af-90ee-4949-be97-f3e6c04416a2-kube-api-access-mxcr6" pod "9328c8af-90ee-4949-be97-f3e6c04416a2" (UID: "9328c8af-90ee-4949-be97-f3e6c04416a2"). InnerVolumeSpecName "kube-api-access-mxcr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:36:06.438834 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.438800 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:06.438834 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.438833 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9328c8af-90ee-4949-be97-f3e6c04416a2-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:06.439034 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.438847 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:06.439034 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.438860 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:06.439034 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.438873 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxcr6\" (UniqueName: \"kubernetes.io/projected/9328c8af-90ee-4949-be97-f3e6c04416a2-kube-api-access-mxcr6\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:06.439034 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.438889 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9328c8af-90ee-4949-be97-f3e6c04416a2-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:06.439034 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.438905 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9328c8af-90ee-4949-be97-f3e6c04416a2-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:06.439034 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.438918 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9328c8af-90ee-4949-be97-f3e6c04416a2-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:06.654676 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.653085 4951 generic.go:356] "Generic (PLEG): container finished" podID="9328c8af-90ee-4949-be97-f3e6c04416a2" containerID="ca0668842600a7ce99755aa25e95e315540eb32a426b75ab9aabfc795bee82a5" exitCode=0 Oct 30 07:36:06.654676 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.653149 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" event={"ID":"9328c8af-90ee-4949-be97-f3e6c04416a2","Type":"ContainerDied","Data":"ca0668842600a7ce99755aa25e95e315540eb32a426b75ab9aabfc795bee82a5"} Oct 30 07:36:06.654676 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.653254 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" event={"ID":"9328c8af-90ee-4949-be97-f3e6c04416a2","Type":"ContainerDied","Data":"f18ca9b4c51be903f14af32b6d6c16916519d33e10d52b9d83509901bcee6d37"} Oct 30 07:36:06.654676 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.653294 4951 scope.go:122] "RemoveContainer" containerID="ca0668842600a7ce99755aa25e95e315540eb32a426b75ab9aabfc795bee82a5" Oct 30 07:36:06.654676 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.653523 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82" Oct 30 07:36:06.680627 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.680593 4951 scope.go:122] "RemoveContainer" containerID="ca0668842600a7ce99755aa25e95e315540eb32a426b75ab9aabfc795bee82a5" Oct 30 07:36:06.680979 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:36:06.680959 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca0668842600a7ce99755aa25e95e315540eb32a426b75ab9aabfc795bee82a5\": container with ID starting with ca0668842600a7ce99755aa25e95e315540eb32a426b75ab9aabfc795bee82a5 not found: ID does not exist" containerID="ca0668842600a7ce99755aa25e95e315540eb32a426b75ab9aabfc795bee82a5" Oct 30 07:36:06.681029 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.680990 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca0668842600a7ce99755aa25e95e315540eb32a426b75ab9aabfc795bee82a5"} err="failed to get container status \"ca0668842600a7ce99755aa25e95e315540eb32a426b75ab9aabfc795bee82a5\": rpc error: code = NotFound desc = could not find container \"ca0668842600a7ce99755aa25e95e315540eb32a426b75ab9aabfc795bee82a5\": container with ID starting with ca0668842600a7ce99755aa25e95e315540eb32a426b75ab9aabfc795bee82a5 not found: ID does not exist" Oct 30 07:36:06.717245 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.717218 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82"] Oct 30 07:36:06.723694 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:06.723670 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4bj82"] Oct 30 07:36:07.773318 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:07.773275 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9328c8af-90ee-4949-be97-f3e6c04416a2" path="/var/lib/kubelet/pods/9328c8af-90ee-4949-be97-f3e6c04416a2/volumes" Oct 30 07:36:18.863415 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:18.860790 4951 generic.go:356] "Generic (PLEG): container finished" podID="9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3" containerID="5c6a8eec409b079ca0e33fab991eb8bbaae4ac44bce0f48415332c5b33d0dbc2" exitCode=0 Oct 30 07:36:18.863415 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:18.861111 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" event={"ID":"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3","Type":"ContainerDied","Data":"5c6a8eec409b079ca0e33fab991eb8bbaae4ac44bce0f48415332c5b33d0dbc2"} Oct 30 07:36:18.863415 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:18.861668 4951 scope.go:122] "RemoveContainer" containerID="5c6a8eec409b079ca0e33fab991eb8bbaae4ac44bce0f48415332c5b33d0dbc2" Oct 30 07:36:18.866272 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:18.866232 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:36:18.866402 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:18.866329 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:36:18.866402 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:18.866380 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:36:18.869172 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:18.869132 4951 generic.go:356] "Generic (PLEG): container finished" podID="bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4" containerID="91a57d17aefb7d1502c0f545ac473c165cd5beada4caf3c52a66d044476b5836" exitCode=0 Oct 30 07:36:18.869172 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:18.869157 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" event={"ID":"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4","Type":"ContainerDied","Data":"91a57d17aefb7d1502c0f545ac473c165cd5beada4caf3c52a66d044476b5836"} Oct 30 07:36:18.869668 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:18.869598 4951 scope.go:122] "RemoveContainer" containerID="91a57d17aefb7d1502c0f545ac473c165cd5beada4caf3c52a66d044476b5836" Oct 30 07:36:18.872829 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:18.871501 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:36:18.872829 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:18.871599 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:36:18.872829 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:18.871645 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:36:19.881177 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:19.881128 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" event={"ID":"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4","Type":"ContainerStarted","Data":"8e3aeacad8d45b21779e48bae2b6239d55a009bbd360118015b2d907cee78997"} Oct 30 07:36:19.883226 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:19.883199 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" event={"ID":"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3","Type":"ContainerStarted","Data":"947900cbb5ceaac48c9cf3052160be0a5312be6fea7b44fc83d757a6ddc0f1ff"} Oct 30 07:36:20.330325 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:20.328876 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:36:20.347595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:20.342726 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 07:36:20.347595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:20.342805 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" podUID="9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 07:36:20.608652 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:20.607643 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:36:20.608652 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:20.607999 4951 patch_prober.go:29] interesting pod/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 07:36:20.608652 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:20.608047 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" podUID="bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 07:36:21.328968 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:21.328928 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 07:36:21.331007 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:21.330970 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" podUID="9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 07:36:21.602414 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:21.602331 4951 patch_prober.go:29] interesting pod/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 07:36:21.602414 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:21.602391 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" podUID="bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 07:36:22.196988 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:22.196952 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf"] Oct 30 07:36:22.197733 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:22.197714 4951 cpu_manager.go:414] "containerMap: removing container" podUID="9328c8af-90ee-4949-be97-f3e6c04416a2" containerName="istio-proxy" Oct 30 07:36:22.197733 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:22.197731 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9328c8af-90ee-4949-be97-f3e6c04416a2" containerName="istio-proxy" Oct 30 07:36:22.197903 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:22.197850 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="9328c8af-90ee-4949-be97-f3e6c04416a2" containerName="istio-proxy" Oct 30 07:36:22.200891 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:22.200873 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf" Oct 30 07:36:22.229364 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:22.229311 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf"] Oct 30 07:36:22.240608 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:22.239946 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-b096ec-serving-cert\"" Oct 30 07:36:22.304609 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:22.302832 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf\" (UID: \"2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf" Oct 30 07:36:22.304609 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:22.302915 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkzvw\" (UniqueName: \"kubernetes.io/projected/2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985-kube-api-access-xkzvw\") pod \"maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf\" (UID: \"2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf" Oct 30 07:36:22.304609 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:22.303215 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985-maas-api-tls\") pod \"maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf\" (UID: \"2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf" Oct 30 07:36:22.304609 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:22.303252 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985-metrics-certs\") pod \"maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf\" (UID: \"2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf" Oct 30 07:36:22.382598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:22.381934 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 07:36:22.382598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:22.382008 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" podUID="9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 07:36:22.406341 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:22.406299 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985-maas-api-tls\") pod \"maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf\" (UID: \"2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf" Oct 30 07:36:22.406341 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:22.406346 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985-metrics-certs\") pod \"maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf\" (UID: \"2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf" Oct 30 07:36:22.406643 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:22.406403 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf\" (UID: \"2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf" Oct 30 07:36:22.406643 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:22.406436 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkzvw\" (UniqueName: \"kubernetes.io/projected/2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985-kube-api-access-xkzvw\") pod \"maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf\" (UID: \"2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf" Oct 30 07:36:22.407601 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:22.407367 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf\" (UID: \"2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf" Oct 30 07:36:22.409650 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:22.409623 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985-metrics-certs\") pod \"maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf\" (UID: \"2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf" Oct 30 07:36:22.410327 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:22.410302 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985-maas-api-tls\") pod \"maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf\" (UID: \"2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf" Oct 30 07:36:22.536648 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:22.438224 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkzvw\" (UniqueName: \"kubernetes.io/projected/2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985-kube-api-access-xkzvw\") pod \"maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf\" (UID: \"2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf" Oct 30 07:36:22.536648 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:22.527884 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf" Oct 30 07:36:22.617668 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:22.616949 4951 patch_prober.go:29] interesting pod/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 07:36:22.617668 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:22.617007 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" podUID="bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 07:36:22.974621 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:22.972688 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:36:23.080643 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:36:23.080324 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:36:23.166597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:23.161838 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf"] Oct 30 07:36:23.173944 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:36:23.173909 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb0ca62_df89_4bd5_bd6a_5be2b3ccb985.slice/crio-3a4aff63fd53d7629ced06fea630689cb6dfd357ce048eb47b7a00e6d356b891 WatchSource:0}: Error finding container 3a4aff63fd53d7629ced06fea630689cb6dfd357ce048eb47b7a00e6d356b891: Status 404 returned error can't find the container with id 3a4aff63fd53d7629ced06fea630689cb6dfd357ce048eb47b7a00e6d356b891 Oct 30 07:36:23.331678 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:23.330865 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 07:36:23.331678 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:23.330928 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" podUID="9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 07:36:23.813203 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:23.812224 4951 patch_prober.go:29] interesting pod/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 07:36:23.813203 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:23.812288 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" podUID="bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 07:36:23.893605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:23.893467 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb"] Oct 30 07:36:23.893605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:23.893541 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v"] Oct 30 07:36:23.893605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:23.893589 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr"] Oct 30 07:36:23.893912 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:23.893796 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" podUID="de840f64-9d0c-4e5b-aa38-19c054fc8b17" containerName="istio-proxy" containerID="cri-o://9c23322f120088c96aa4b1eb87191910c7dde6908aae780249ca9cbb3d9b9ac6" gracePeriod=30 Oct 30 07:36:23.894830 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:23.894065 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" podUID="f0bcd71c-9f82-4f64-81f0-1c343709cfc6" containerName="istio-proxy" containerID="cri-o://b0511698b97e751f422a33a92ac77ebebd7071af46246af28f8ffc719f46b1c2" gracePeriod=30 Oct 30 07:36:23.894830 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:23.894250 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" podUID="f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5" containerName="istio-proxy" containerID="cri-o://6246129b09d8e1fb501eff895a819d035277c421b6cd4ca898a9c8962eab99a4" gracePeriod=30 Oct 30 07:36:23.940600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:23.920432 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf" event={"ID":"2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985","Type":"ContainerStarted","Data":"3a4aff63fd53d7629ced06fea630689cb6dfd357ce048eb47b7a00e6d356b891"} Oct 30 07:36:24.351596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:24.330598 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 07:36:24.351596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:24.330655 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" podUID="9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 07:36:24.663649 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:24.662511 4951 patch_prober.go:29] interesting pod/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 07:36:24.663649 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:24.662590 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" podUID="bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 07:36:24.968037 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:24.967926 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf" event={"ID":"2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985","Type":"ContainerStarted","Data":"26de1a58e5970b358caa8c468faa8d53799278e1664e49343d996487fbd5a423"} Oct 30 07:36:24.970213 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:24.970181 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf" Oct 30 07:36:25.019187 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:25.019107 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf" podStartSLOduration=2.603380698 podStartE2EDuration="3.019092521s" totalImagesPullingTime="415.711823ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:36:25.017344577 +0000 UTC m=+2391.360392544" watchObservedRunningTime="2026-10-30 07:36:25.019092521 +0000 UTC m=+2391.362140485" Oct 30 07:36:25.349595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:25.349008 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 07:36:25.349595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:25.349077 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" podUID="9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 07:36:25.602947 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:25.602842 4951 patch_prober.go:29] interesting pod/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 07:36:25.602947 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:25.602908 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" podUID="bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 07:36:26.173118 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:26.173082 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:36:26.332809 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:26.332772 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:36:26.334122 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:26.334098 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 07:36:26.607659 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:26.607629 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:36:27.000895 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:27.000860 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 07:36:29.021801 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.021769 4951 generic.go:356] "Generic (PLEG): container finished" podID="f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5" containerID="6246129b09d8e1fb501eff895a819d035277c421b6cd4ca898a9c8962eab99a4" exitCode=0 Oct 30 07:36:29.022215 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.021978 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" event={"ID":"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5","Type":"ContainerDied","Data":"6246129b09d8e1fb501eff895a819d035277c421b6cd4ca898a9c8962eab99a4"} Oct 30 07:36:29.276786 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.276755 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:36:29.437753 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.435114 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istio-envoy\") pod \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " Oct 30 07:36:29.437753 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.435167 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-workload-certs\") pod \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " Oct 30 07:36:29.437753 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.435235 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-credential-socket\") pod \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " Oct 30 07:36:29.437753 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.435269 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istiod-ca-cert\") pod \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " Oct 30 07:36:29.437753 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.435361 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-workload-socket\") pod \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " Oct 30 07:36:29.437753 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.435552 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istio-podinfo\") pod \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " Oct 30 07:36:29.437753 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.435610 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-kube-api-access-7ctlr\" (UniqueName: \"kubernetes.io/projected/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-kube-api-access-7ctlr\") pod \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " Oct 30 07:36:29.437753 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.435670 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istio-token\" (UniqueName: \"kubernetes.io/projected/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istio-token\") pod \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " Oct 30 07:36:29.437753 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.435773 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istio-data\") pod \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\" (UID: \"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5\") " Oct 30 07:36:29.445926 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.445347 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-workload-socket" pod "f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5" (UID: "f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:29.447382 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.447352 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istiod-ca-cert" pod "f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5" (UID: "f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:36:29.460554 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.458987 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istio-data" pod "f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5" (UID: "f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:29.460554 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.459197 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-credential-socket" pod "f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5" (UID: "f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:29.460554 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.459775 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-workload-certs" pod "f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5" (UID: "f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:29.462013 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.461959 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istio-token" pod "f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5" (UID: "f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:36:29.463044 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.462561 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istio-podinfo" pod "f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5" (UID: "f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:36:29.464071 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.464041 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-kube-api-access-7ctlr" pod "f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5" (UID: "f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5"). InnerVolumeSpecName "kube-api-access-7ctlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:36:29.466389 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.466352 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istio-envoy" pod "f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5" (UID: "f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:29.467297 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.466961 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:36:29.493143 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.493122 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:36:29.537292 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.537258 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istiod-ca-cert\") pod \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " Oct 30 07:36:29.539185 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.539160 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-kube-api-access-zdtj5\" (UniqueName: \"kubernetes.io/projected/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-kube-api-access-zdtj5\") pod \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " Oct 30 07:36:29.539437 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.539395 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istio-podinfo\") pod \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " Oct 30 07:36:29.539437 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.538293 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istiod-ca-cert" pod "f0bcd71c-9f82-4f64-81f0-1c343709cfc6" (UID: "f0bcd71c-9f82-4f64-81f0-1c343709cfc6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:36:29.539634 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.539443 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istio-token\" (UniqueName: \"kubernetes.io/projected/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istio-token\") pod \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " Oct 30 07:36:29.539634 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.539488 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istio-data\") pod \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " Oct 30 07:36:29.539634 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.539562 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-workload-certs\") pod \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " Oct 30 07:36:29.539794 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.539757 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-workload-socket\") pod \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " Oct 30 07:36:29.539851 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.539800 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istio-envoy\") pod \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " Oct 30 07:36:29.539992 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.539977 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-credential-socket\") pod \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\" (UID: \"f0bcd71c-9f82-4f64-81f0-1c343709cfc6\") " Oct 30 07:36:29.541118 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.540592 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-credential-socket" pod "f0bcd71c-9f82-4f64-81f0-1c343709cfc6" (UID: "f0bcd71c-9f82-4f64-81f0-1c343709cfc6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:29.541118 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.540729 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-workload-certs" pod "f0bcd71c-9f82-4f64-81f0-1c343709cfc6" (UID: "f0bcd71c-9f82-4f64-81f0-1c343709cfc6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:29.541358 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.540825 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istio-data" pod "f0bcd71c-9f82-4f64-81f0-1c343709cfc6" (UID: "f0bcd71c-9f82-4f64-81f0-1c343709cfc6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:29.541358 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.540835 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-workload-socket" pod "f0bcd71c-9f82-4f64-81f0-1c343709cfc6" (UID: "f0bcd71c-9f82-4f64-81f0-1c343709cfc6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:29.542049 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.542029 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:29.542180 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.542166 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:29.542267 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.542256 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:29.542360 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.542349 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:29.542457 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.542446 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:29.542555 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.542526 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:29.542724 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.542709 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:29.542822 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.542806 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:29.542913 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.542900 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:29.542999 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.542988 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:29.543098 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.543085 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:29.543191 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.543178 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:29.543280 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.543269 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ctlr\" (UniqueName: \"kubernetes.io/projected/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-kube-api-access-7ctlr\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:29.543367 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.543356 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:29.543755 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.543725 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-kube-api-access-zdtj5" pod "f0bcd71c-9f82-4f64-81f0-1c343709cfc6" (UID: "f0bcd71c-9f82-4f64-81f0-1c343709cfc6"). InnerVolumeSpecName "kube-api-access-zdtj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:36:29.544357 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.544337 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istio-podinfo" pod "f0bcd71c-9f82-4f64-81f0-1c343709cfc6" (UID: "f0bcd71c-9f82-4f64-81f0-1c343709cfc6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:36:29.548640 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.547271 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istio-token" pod "f0bcd71c-9f82-4f64-81f0-1c343709cfc6" (UID: "f0bcd71c-9f82-4f64-81f0-1c343709cfc6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:36:29.548760 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.548738 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istio-envoy" pod "f0bcd71c-9f82-4f64-81f0-1c343709cfc6" (UID: "f0bcd71c-9f82-4f64-81f0-1c343709cfc6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:29.644161 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.644130 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-workload-socket\") pod \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " Oct 30 07:36:29.644328 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.644188 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istio-envoy\") pod \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " Oct 30 07:36:29.644328 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.644288 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istio-data\") pod \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " Oct 30 07:36:29.644447 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.644384 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istio-podinfo\") pod \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " Oct 30 07:36:29.644447 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.644427 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de840f64-9d0c-4e5b-aa38-19c054fc8b17-kube-api-access-wdbph\" (UniqueName: \"kubernetes.io/projected/de840f64-9d0c-4e5b-aa38-19c054fc8b17-kube-api-access-wdbph\") pod \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " Oct 30 07:36:29.644562 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.644459 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-workload-socket" pod "de840f64-9d0c-4e5b-aa38-19c054fc8b17" (UID: "de840f64-9d0c-4e5b-aa38-19c054fc8b17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:29.644672 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.644549 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-credential-socket\") pod \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " Oct 30 07:36:29.644672 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.644646 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istio-data" pod "de840f64-9d0c-4e5b-aa38-19c054fc8b17" (UID: "de840f64-9d0c-4e5b-aa38-19c054fc8b17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:29.644672 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.644663 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istiod-ca-cert\") pod \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " Oct 30 07:36:29.644848 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.644698 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-workload-certs\") pod \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " Oct 30 07:36:29.644848 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.644760 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istio-token\" (UniqueName: \"kubernetes.io/projected/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istio-token\") pod \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\" (UID: \"de840f64-9d0c-4e5b-aa38-19c054fc8b17\") " Oct 30 07:36:29.644979 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.644852 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-credential-socket" pod "de840f64-9d0c-4e5b-aa38-19c054fc8b17" (UID: "de840f64-9d0c-4e5b-aa38-19c054fc8b17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:29.644979 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.644950 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istiod-ca-cert" pod "de840f64-9d0c-4e5b-aa38-19c054fc8b17" (UID: "de840f64-9d0c-4e5b-aa38-19c054fc8b17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:36:29.645131 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.645093 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-workload-certs" pod "de840f64-9d0c-4e5b-aa38-19c054fc8b17" (UID: "de840f64-9d0c-4e5b-aa38-19c054fc8b17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:29.645495 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.645478 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:29.645641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.645501 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:29.645641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.645515 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:29.645641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.645528 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:29.645641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.645546 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:29.645641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.645558 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdtj5\" (UniqueName: \"kubernetes.io/projected/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-kube-api-access-zdtj5\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:29.645641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.645586 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:29.645641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.645601 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:29.645641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.645621 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0bcd71c-9f82-4f64-81f0-1c343709cfc6-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:29.646885 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.646848 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istio-podinfo" pod "de840f64-9d0c-4e5b-aa38-19c054fc8b17" (UID: "de840f64-9d0c-4e5b-aa38-19c054fc8b17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:36:29.647283 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.647262 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istio-token" pod "de840f64-9d0c-4e5b-aa38-19c054fc8b17" (UID: "de840f64-9d0c-4e5b-aa38-19c054fc8b17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:36:29.647283 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.647274 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istio-envoy" pod "de840f64-9d0c-4e5b-aa38-19c054fc8b17" (UID: "de840f64-9d0c-4e5b-aa38-19c054fc8b17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:29.647844 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.647816 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de840f64-9d0c-4e5b-aa38-19c054fc8b17-kube-api-access-wdbph" pod "de840f64-9d0c-4e5b-aa38-19c054fc8b17" (UID: "de840f64-9d0c-4e5b-aa38-19c054fc8b17"). InnerVolumeSpecName "kube-api-access-wdbph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:36:29.747604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.747500 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:29.747604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.747542 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdbph\" (UniqueName: \"kubernetes.io/projected/de840f64-9d0c-4e5b-aa38-19c054fc8b17-kube-api-access-wdbph\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:29.747604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.747553 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:29.747604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:29.747562 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de840f64-9d0c-4e5b-aa38-19c054fc8b17-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:36:30.038186 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:30.038144 4951 generic.go:356] "Generic (PLEG): container finished" podID="de840f64-9d0c-4e5b-aa38-19c054fc8b17" containerID="9c23322f120088c96aa4b1eb87191910c7dde6908aae780249ca9cbb3d9b9ac6" exitCode=0 Oct 30 07:36:30.039080 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:30.039047 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" Oct 30 07:36:30.040341 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:30.039097 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" event={"ID":"de840f64-9d0c-4e5b-aa38-19c054fc8b17","Type":"ContainerDied","Data":"9c23322f120088c96aa4b1eb87191910c7dde6908aae780249ca9cbb3d9b9ac6"} Oct 30 07:36:30.040341 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:30.039513 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr" event={"ID":"de840f64-9d0c-4e5b-aa38-19c054fc8b17","Type":"ContainerDied","Data":"7d4a5d33096adce750b56bbee136e3d55ef543e1941ebf87acf479d35d714502"} Oct 30 07:36:30.040341 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:30.039608 4951 scope.go:122] "RemoveContainer" containerID="9c23322f120088c96aa4b1eb87191910c7dde6908aae780249ca9cbb3d9b9ac6" Oct 30 07:36:30.041698 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:30.041225 4951 generic.go:356] "Generic (PLEG): container finished" podID="f0bcd71c-9f82-4f64-81f0-1c343709cfc6" containerID="b0511698b97e751f422a33a92ac77ebebd7071af46246af28f8ffc719f46b1c2" exitCode=0 Oct 30 07:36:30.041698 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:30.041354 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" event={"ID":"f0bcd71c-9f82-4f64-81f0-1c343709cfc6","Type":"ContainerDied","Data":"b0511698b97e751f422a33a92ac77ebebd7071af46246af28f8ffc719f46b1c2"} Oct 30 07:36:30.041698 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:30.041407 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" event={"ID":"f0bcd71c-9f82-4f64-81f0-1c343709cfc6","Type":"ContainerDied","Data":"c1845a9751d57bf5ffd91d5a10b90665430f7e962cd551c8edd7c1c02f788f4c"} Oct 30 07:36:30.041698 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:30.041512 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" Oct 30 07:36:30.045696 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:30.045664 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" event={"ID":"f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5","Type":"ContainerDied","Data":"583bb7c3b010a5be0e705cf0c674cddd199a6d76347df55bd1ed54ae78e3e403"} Oct 30 07:36:30.045835 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:30.045674 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v" Oct 30 07:36:30.055060 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:30.055034 4951 scope.go:122] "RemoveContainer" containerID="b71ecc855f084d3301e8e09f292911517600b11035063e209cff52d778f237c0" Oct 30 07:36:30.066033 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:30.066007 4951 scope.go:122] "RemoveContainer" containerID="9c23322f120088c96aa4b1eb87191910c7dde6908aae780249ca9cbb3d9b9ac6" Oct 30 07:36:30.066423 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:36:30.066396 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c23322f120088c96aa4b1eb87191910c7dde6908aae780249ca9cbb3d9b9ac6\": container with ID starting with 9c23322f120088c96aa4b1eb87191910c7dde6908aae780249ca9cbb3d9b9ac6 not found: ID does not exist" containerID="9c23322f120088c96aa4b1eb87191910c7dde6908aae780249ca9cbb3d9b9ac6" Oct 30 07:36:30.066504 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:30.066435 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c23322f120088c96aa4b1eb87191910c7dde6908aae780249ca9cbb3d9b9ac6"} err="failed to get container status \"9c23322f120088c96aa4b1eb87191910c7dde6908aae780249ca9cbb3d9b9ac6\": rpc error: code = NotFound desc = could not find container \"9c23322f120088c96aa4b1eb87191910c7dde6908aae780249ca9cbb3d9b9ac6\": container with ID starting with 9c23322f120088c96aa4b1eb87191910c7dde6908aae780249ca9cbb3d9b9ac6 not found: ID does not exist" Oct 30 07:36:30.066504 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:30.066455 4951 scope.go:122] "RemoveContainer" containerID="b71ecc855f084d3301e8e09f292911517600b11035063e209cff52d778f237c0" Oct 30 07:36:30.066784 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:36:30.066769 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b71ecc855f084d3301e8e09f292911517600b11035063e209cff52d778f237c0\": container with ID starting with b71ecc855f084d3301e8e09f292911517600b11035063e209cff52d778f237c0 not found: ID does not exist" containerID="b71ecc855f084d3301e8e09f292911517600b11035063e209cff52d778f237c0" Oct 30 07:36:30.066845 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:30.066791 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b71ecc855f084d3301e8e09f292911517600b11035063e209cff52d778f237c0"} err="failed to get container status \"b71ecc855f084d3301e8e09f292911517600b11035063e209cff52d778f237c0\": rpc error: code = NotFound desc = could not find container \"b71ecc855f084d3301e8e09f292911517600b11035063e209cff52d778f237c0\": container with ID starting with b71ecc855f084d3301e8e09f292911517600b11035063e209cff52d778f237c0 not found: ID does not exist" Oct 30 07:36:30.066845 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:30.066809 4951 scope.go:122] "RemoveContainer" containerID="b0511698b97e751f422a33a92ac77ebebd7071af46246af28f8ffc719f46b1c2" Oct 30 07:36:30.076724 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:30.076699 4951 scope.go:122] "RemoveContainer" containerID="b0511698b97e751f422a33a92ac77ebebd7071af46246af28f8ffc719f46b1c2" Oct 30 07:36:30.077029 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:36:30.077009 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0511698b97e751f422a33a92ac77ebebd7071af46246af28f8ffc719f46b1c2\": container with ID starting with b0511698b97e751f422a33a92ac77ebebd7071af46246af28f8ffc719f46b1c2 not found: ID does not exist" containerID="b0511698b97e751f422a33a92ac77ebebd7071af46246af28f8ffc719f46b1c2" Oct 30 07:36:30.077106 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:30.077045 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0511698b97e751f422a33a92ac77ebebd7071af46246af28f8ffc719f46b1c2"} err="failed to get container status \"b0511698b97e751f422a33a92ac77ebebd7071af46246af28f8ffc719f46b1c2\": rpc error: code = NotFound desc = could not find container \"b0511698b97e751f422a33a92ac77ebebd7071af46246af28f8ffc719f46b1c2\": container with ID starting with b0511698b97e751f422a33a92ac77ebebd7071af46246af28f8ffc719f46b1c2 not found: ID does not exist" Oct 30 07:36:30.077106 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:30.077065 4951 scope.go:122] "RemoveContainer" containerID="6246129b09d8e1fb501eff895a819d035277c421b6cd4ca898a9c8962eab99a4" Oct 30 07:36:30.098100 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:30.098074 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr"] Oct 30 07:36:30.105097 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:30.105072 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-j6htr"] Oct 30 07:36:30.122729 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:30.122698 4951 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:36:30.122865 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:30.122746 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb" podUID="f0bcd71c-9f82-4f64-81f0-1c343709cfc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:36:30.143362 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:30.143330 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb"] Oct 30 07:36:30.148627 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:30.148600 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-7t2wb"] Oct 30 07:36:30.186811 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:30.186778 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v"] Oct 30 07:36:30.192234 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:30.192206 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-kdd9v"] Oct 30 07:36:31.775311 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:31.775276 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de840f64-9d0c-4e5b-aa38-19c054fc8b17" path="/var/lib/kubelet/pods/de840f64-9d0c-4e5b-aa38-19c054fc8b17/volumes" Oct 30 07:36:31.775980 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:31.775963 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0bcd71c-9f82-4f64-81f0-1c343709cfc6" path="/var/lib/kubelet/pods/f0bcd71c-9f82-4f64-81f0-1c343709cfc6/volumes" Oct 30 07:36:31.776409 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:31.776397 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5" path="/var/lib/kubelet/pods/f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5/volumes" Oct 30 07:36:32.006127 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:32.006097 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf" Oct 30 07:36:34.450538 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:34.450506 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/3.log" Oct 30 07:36:34.456485 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:34.456458 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 07:36:34.458713 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:34.458689 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/3.log" Oct 30 07:36:34.464626 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:34.464605 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 07:36:35.841493 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:35.841456 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p"] Oct 30 07:36:35.842627 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:35.842601 4951 cpu_manager.go:414] "containerMap: removing container" podUID="f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5" containerName="istio-proxy" Oct 30 07:36:35.842627 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:35.842626 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5" containerName="istio-proxy" Oct 30 07:36:35.842825 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:35.842661 4951 cpu_manager.go:414] "containerMap: removing container" podUID="f0bcd71c-9f82-4f64-81f0-1c343709cfc6" containerName="istio-proxy" Oct 30 07:36:35.842825 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:35.842669 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0bcd71c-9f82-4f64-81f0-1c343709cfc6" containerName="istio-proxy" Oct 30 07:36:35.842825 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:35.842721 4951 cpu_manager.go:414] "containerMap: removing container" podUID="de840f64-9d0c-4e5b-aa38-19c054fc8b17" containerName="istio-proxy" Oct 30 07:36:35.842825 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:35.842728 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de840f64-9d0c-4e5b-aa38-19c054fc8b17" containerName="istio-proxy" Oct 30 07:36:35.842825 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:35.842750 4951 cpu_manager.go:414] "containerMap: removing container" podUID="de840f64-9d0c-4e5b-aa38-19c054fc8b17" containerName="istio-proxy" Oct 30 07:36:35.842825 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:35.842757 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de840f64-9d0c-4e5b-aa38-19c054fc8b17" containerName="istio-proxy" Oct 30 07:36:35.843201 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:35.842893 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6af01ce-6c4d-45a0-94a3-0ed2a9ec7bd5" containerName="istio-proxy" Oct 30 07:36:35.843201 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:35.842909 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0bcd71c-9f82-4f64-81f0-1c343709cfc6" containerName="istio-proxy" Oct 30 07:36:35.843201 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:35.842924 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="de840f64-9d0c-4e5b-aa38-19c054fc8b17" containerName="istio-proxy" Oct 30 07:36:35.843201 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:35.842955 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="de840f64-9d0c-4e5b-aa38-19c054fc8b17" containerName="istio-proxy" Oct 30 07:36:35.849323 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:35.849294 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:36:35.872433 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:35.872381 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-a91a83-openshift-default-dockercfg-gn4nj\"" Oct 30 07:36:35.889133 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:35.889094 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p"] Oct 30 07:36:35.979778 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:35.953447 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c9697a-7dea-4662-8da4-c2618945d160-istiod-ca-cert\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:36:36.003921 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:36.001894 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c9697a-7dea-4662-8da4-c2618945d160-istio-podinfo\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:36:36.003921 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:36.002089 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-istio-data\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:36:36.003921 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:36.002226 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffg47\" (UniqueName: \"kubernetes.io/projected/d0c9697a-7dea-4662-8da4-c2618945d160-kube-api-access-ffg47\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:36:36.003921 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:36.002262 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-workload-certs\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:36:36.003921 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:36.002321 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-istio-envoy\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:36:36.003921 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:36.002364 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-credential-socket\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:36:36.003921 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:36.002417 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-workload-socket\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:36:36.003921 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:36.002542 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c9697a-7dea-4662-8da4-c2618945d160-istio-token\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:36:36.111640 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:36.107829 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-istio-data\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:36:36.111640 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:36.107933 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffg47\" (UniqueName: \"kubernetes.io/projected/d0c9697a-7dea-4662-8da4-c2618945d160-kube-api-access-ffg47\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:36:36.111640 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:36.107966 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-workload-certs\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:36:36.111640 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:36.108017 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-istio-envoy\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:36:36.111640 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:36.108057 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-credential-socket\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:36:36.111640 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:36.108107 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-workload-socket\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:36:36.111640 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:36.108208 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c9697a-7dea-4662-8da4-c2618945d160-istio-token\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:36:36.111640 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:36.108278 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c9697a-7dea-4662-8da4-c2618945d160-istiod-ca-cert\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:36:36.111640 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:36.108331 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c9697a-7dea-4662-8da4-c2618945d160-istio-podinfo\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:36:36.114593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:36.114548 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-istio-data\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:36:36.119819 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:36.114897 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-workload-socket\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:36:36.119966 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:36.115125 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-credential-socket\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:36:36.120050 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:36.115331 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-workload-certs\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:36:36.120133 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:36.116214 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c9697a-7dea-4662-8da4-c2618945d160-istiod-ca-cert\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:36:36.124835 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:36.124803 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c9697a-7dea-4662-8da4-c2618945d160-istio-podinfo\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:36:36.128857 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:36.128833 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-istio-envoy\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:36:36.155873 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:36.153909 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffg47\" (UniqueName: \"kubernetes.io/projected/d0c9697a-7dea-4662-8da4-c2618945d160-kube-api-access-ffg47\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:36:36.168825 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:36.168792 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c9697a-7dea-4662-8da4-c2618945d160-istio-token\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:36:36.175339 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:36.175300 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:36:36.549720 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:36.549524 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p"] Oct 30 07:36:36.551391 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:36:36.551354 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c9697a_7dea_4662_8da4_c2618945d160.slice/crio-158308feb10bee4ef6af6de8442674ba0f33b2fa34e42b066bda34421c25b5e1 WatchSource:0}: Error finding container 158308feb10bee4ef6af6de8442674ba0f33b2fa34e42b066bda34421c25b5e1: Status 404 returned error can't find the container with id 158308feb10bee4ef6af6de8442674ba0f33b2fa34e42b066bda34421c25b5e1 Oct 30 07:36:36.555684 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:36.555627 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:36:36.555808 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:36.555734 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:36:36.555808 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:36.555783 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:36:37.120135 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:37.120100 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" event={"ID":"d0c9697a-7dea-4662-8da4-c2618945d160","Type":"ContainerStarted","Data":"4c0dcfe40ef7c95e5594b253f38058adc8af6f1c0156d183282f1d07aa052521"} Oct 30 07:36:37.120541 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:37.120152 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" event={"ID":"d0c9697a-7dea-4662-8da4-c2618945d160","Type":"ContainerStarted","Data":"158308feb10bee4ef6af6de8442674ba0f33b2fa34e42b066bda34421c25b5e1"} Oct 30 07:36:37.176492 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:37.176461 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:36:37.182894 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:37.182870 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:36:37.199453 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:37.199397 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" podStartSLOduration=2.199384271 podStartE2EDuration="2.199384271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:36:37.183185666 +0000 UTC m=+2403.526233653" watchObservedRunningTime="2026-10-30 07:36:37.199384271 +0000 UTC m=+2403.542432235" Oct 30 07:36:38.127377 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:38.127347 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:36:38.128235 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:38.128218 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:36:38.583926 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:38.583896 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7"] Oct 30 07:36:38.588797 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:38.588775 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:36:38.609438 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:38.609390 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-bffe43-openshift-default-dockercfg-nxzk7\"" Oct 30 07:36:38.630167 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:38.630123 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7"] Oct 30 07:36:38.656373 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:38.656333 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-workload-socket\") pod \"e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:36:38.656899 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:38.656870 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-workload-certs\") pod \"e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:36:38.657097 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:38.657079 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-istio-envoy\") pod \"e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:36:38.657256 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:38.657240 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f52156d3-44fc-4983-984c-8ac36b1cb942-istiod-ca-cert\") pod \"e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:36:38.657392 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:38.657374 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2rw7\" (UniqueName: \"kubernetes.io/projected/f52156d3-44fc-4983-984c-8ac36b1cb942-kube-api-access-j2rw7\") pod \"e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:36:38.657670 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:38.657650 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f52156d3-44fc-4983-984c-8ac36b1cb942-istio-token\") pod \"e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:36:38.657821 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:38.657807 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f52156d3-44fc-4983-984c-8ac36b1cb942-istio-podinfo\") pod \"e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:36:38.658002 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:38.657987 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-credential-socket\") pod \"e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:36:38.658231 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:38.658206 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-istio-data\") pod \"e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:36:38.768594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:38.767846 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-workload-socket\") pod \"e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:36:38.768594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:38.767912 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-workload-certs\") pod \"e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:36:38.768594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:38.767948 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-istio-envoy\") pod \"e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:36:38.768594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:38.767984 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f52156d3-44fc-4983-984c-8ac36b1cb942-istiod-ca-cert\") pod \"e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:36:38.768594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:38.768014 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2rw7\" (UniqueName: \"kubernetes.io/projected/f52156d3-44fc-4983-984c-8ac36b1cb942-kube-api-access-j2rw7\") pod \"e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:36:38.768594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:38.768099 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f52156d3-44fc-4983-984c-8ac36b1cb942-istio-token\") pod \"e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:36:38.768594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:38.768140 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f52156d3-44fc-4983-984c-8ac36b1cb942-istio-podinfo\") pod \"e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:36:38.768594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:38.768203 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-credential-socket\") pod \"e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:36:38.768594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:38.768247 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-istio-data\") pod \"e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:36:38.798596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:38.797597 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f52156d3-44fc-4983-984c-8ac36b1cb942-istiod-ca-cert\") pod \"e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:36:38.798596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:38.797920 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-workload-socket\") pod \"e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:36:38.798596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:38.798172 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-workload-certs\") pod \"e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:36:38.809779 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:38.799284 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-credential-socket\") pod \"e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:36:38.809779 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:38.807774 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-istio-envoy\") pod \"e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:36:38.818379 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:38.818254 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-istio-data\") pod \"e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:36:38.894651 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:38.885995 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f52156d3-44fc-4983-984c-8ac36b1cb942-istio-podinfo\") pod \"e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:36:38.894651 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:38.891482 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f52156d3-44fc-4983-984c-8ac36b1cb942-istio-token\") pod \"e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:36:38.930650 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:38.920311 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2rw7\" (UniqueName: \"kubernetes.io/projected/f52156d3-44fc-4983-984c-8ac36b1cb942-kube-api-access-j2rw7\") pod \"e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:36:39.209535 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:39.209443 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:36:39.543472 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:36:39.543440 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf52156d3_44fc_4983_984c_8ac36b1cb942.slice/crio-f41a46da4d7b2f8f2fd81f5b38c94ad495bf4a38c7ceeeca4b0af627f61f33d5 WatchSource:0}: Error finding container f41a46da4d7b2f8f2fd81f5b38c94ad495bf4a38c7ceeeca4b0af627f61f33d5: Status 404 returned error can't find the container with id f41a46da4d7b2f8f2fd81f5b38c94ad495bf4a38c7ceeeca4b0af627f61f33d5 Oct 30 07:36:39.547107 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:39.547070 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:36:39.547197 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:39.547144 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:36:39.547197 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:39.547172 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:36:39.547536 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:39.547502 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7"] Oct 30 07:36:40.151507 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:40.151463 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" event={"ID":"f52156d3-44fc-4983-984c-8ac36b1cb942","Type":"ContainerStarted","Data":"0d56edeb5190f4570765bd8943b7ca7c92294d37c62e21b5831dcb036331a833"} Oct 30 07:36:40.151721 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:40.151523 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" event={"ID":"f52156d3-44fc-4983-984c-8ac36b1cb942","Type":"ContainerStarted","Data":"f41a46da4d7b2f8f2fd81f5b38c94ad495bf4a38c7ceeeca4b0af627f61f33d5"} Oct 30 07:36:40.209867 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:40.209821 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:36:40.214655 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:40.214595 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" podStartSLOduration=2.214559396 podStartE2EDuration="2.214559396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:36:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:36:40.213677351 +0000 UTC m=+2406.556725343" watchObservedRunningTime="2026-10-30 07:36:40.214559396 +0000 UTC m=+2406.557607362" Oct 30 07:36:40.218466 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:40.218436 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:36:41.158883 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:41.158846 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:36:41.159699 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:41.159677 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:36:44.713237 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:44.713188 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb"] Oct 30 07:36:44.719177 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:44.719150 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:36:44.763594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:44.762817 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-1fb038-openshift-default-dockercfg-wz87s\"" Oct 30 07:36:44.777470 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:44.777442 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb"] Oct 30 07:36:44.816087 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:44.816053 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-istio-envoy\") pod \"e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:36:44.816281 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:44.816259 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvwr8\" (UniqueName: \"kubernetes.io/projected/a398df37-12de-4a79-a5cb-d246f85cfa5f-kube-api-access-pvwr8\") pod \"e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:36:44.816433 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:44.816305 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-credential-socket\") pod \"e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:36:44.816433 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:44.816328 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-istio-data\") pod \"e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:36:44.816554 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:44.816454 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a398df37-12de-4a79-a5cb-d246f85cfa5f-istiod-ca-cert\") pod \"e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:36:44.816554 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:44.816500 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-workload-socket\") pod \"e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:36:44.816685 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:44.816638 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a398df37-12de-4a79-a5cb-d246f85cfa5f-istio-token\") pod \"e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:36:44.816685 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:44.816672 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-workload-certs\") pod \"e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:36:44.816803 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:44.816743 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a398df37-12de-4a79-a5cb-d246f85cfa5f-istio-podinfo\") pod \"e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:36:44.918550 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:44.918494 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvwr8\" (UniqueName: \"kubernetes.io/projected/a398df37-12de-4a79-a5cb-d246f85cfa5f-kube-api-access-pvwr8\") pod \"e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:36:44.918940 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:44.918903 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-credential-socket\") pod \"e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:36:44.919169 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:44.919147 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-istio-data\") pod \"e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:36:44.919347 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:44.919332 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a398df37-12de-4a79-a5cb-d246f85cfa5f-istiod-ca-cert\") pod \"e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:36:44.919488 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:44.919472 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-workload-socket\") pod \"e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:36:44.919659 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:44.919642 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a398df37-12de-4a79-a5cb-d246f85cfa5f-istio-token\") pod \"e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:36:44.919830 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:44.919809 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-workload-certs\") pod \"e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:36:44.920038 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:44.920021 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a398df37-12de-4a79-a5cb-d246f85cfa5f-istio-podinfo\") pod \"e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:36:44.920287 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:44.920273 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-istio-envoy\") pod \"e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:36:44.940598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:44.923285 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-credential-socket\") pod \"e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:36:44.940598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:44.923530 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-istio-data\") pod \"e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:36:44.940598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:44.940140 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-workload-socket\") pod \"e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:36:44.940959 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:44.940824 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-workload-certs\") pod \"e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:36:44.942797 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:44.942731 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a398df37-12de-4a79-a5cb-d246f85cfa5f-istiod-ca-cert\") pod \"e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:36:44.980387 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:44.980307 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-istio-envoy\") pod \"e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:36:44.988936 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:44.988329 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvwr8\" (UniqueName: \"kubernetes.io/projected/a398df37-12de-4a79-a5cb-d246f85cfa5f-kube-api-access-pvwr8\") pod \"e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:36:44.988936 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:44.988880 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a398df37-12de-4a79-a5cb-d246f85cfa5f-istio-podinfo\") pod \"e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:36:45.020788 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:45.010157 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a398df37-12de-4a79-a5cb-d246f85cfa5f-istio-token\") pod \"e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:36:45.080680 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:45.080628 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:36:45.466458 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:36:45.466426 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda398df37_12de_4a79_a5cb_d246f85cfa5f.slice/crio-5635791462f1938ce1029de2c2dd84cacae9ebec1bd3357d33849009ad8fe054 WatchSource:0}: Error finding container 5635791462f1938ce1029de2c2dd84cacae9ebec1bd3357d33849009ad8fe054: Status 404 returned error can't find the container with id 5635791462f1938ce1029de2c2dd84cacae9ebec1bd3357d33849009ad8fe054 Oct 30 07:36:45.468740 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:45.468709 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb"] Oct 30 07:36:45.469385 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:45.469350 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:36:45.469447 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:45.469417 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:36:45.469486 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:45.469446 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:36:46.215555 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:46.215512 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" event={"ID":"a398df37-12de-4a79-a5cb-d246f85cfa5f","Type":"ContainerStarted","Data":"670feba0db6aae9046b30f93d28b62fb77dfecf02e91fcf4a14926b073ec6395"} Oct 30 07:36:46.216001 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:46.215564 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" event={"ID":"a398df37-12de-4a79-a5cb-d246f85cfa5f","Type":"ContainerStarted","Data":"5635791462f1938ce1029de2c2dd84cacae9ebec1bd3357d33849009ad8fe054"} Oct 30 07:36:46.289812 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:46.289755 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" podStartSLOduration=2.289739969 podStartE2EDuration="2.289739969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:36:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:36:46.280652406 +0000 UTC m=+2412.623700370" watchObservedRunningTime="2026-10-30 07:36:46.289739969 +0000 UTC m=+2412.632787933" Oct 30 07:36:47.081586 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:47.081547 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:36:47.088590 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:47.088541 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:36:47.244597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:47.237972 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:36:47.271603 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:47.262785 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:36:51.512741 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:51.512698 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:36:51.517773 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:51.512760 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" podUID="3ad4d41c-7af3-453d-9005-d34264ed3acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:36:52.280203 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:52.280167 4951 generic.go:356] "Generic (PLEG): container finished" podID="3ad4d41c-7af3-453d-9005-d34264ed3acd" containerID="818324153ca59fed54d04cb66f35f25f4dcb53b96e013b9b2fa0690353ed7c60" exitCode=0 Oct 30 07:36:52.280395 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:52.280245 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" event={"ID":"3ad4d41c-7af3-453d-9005-d34264ed3acd","Type":"ContainerDied","Data":"818324153ca59fed54d04cb66f35f25f4dcb53b96e013b9b2fa0690353ed7c60"} Oct 30 07:36:52.280790 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:52.280773 4951 scope.go:122] "RemoveContainer" containerID="818324153ca59fed54d04cb66f35f25f4dcb53b96e013b9b2fa0690353ed7c60" Oct 30 07:36:52.282720 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:52.282689 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:36:52.282808 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:52.282764 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:36:52.282808 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:52.282795 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:36:53.027295 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:36:53.027244 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:36:53.289004 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:53.288918 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" event={"ID":"3ad4d41c-7af3-453d-9005-d34264ed3acd","Type":"ContainerStarted","Data":"a44624c74aedb78858324a1d5e5f3bdd943559c48d8d2a150faf5ea9ceb8822a"} Oct 30 07:36:53.739662 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:53.729996 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:36:53.739662 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:53.730138 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 30 07:36:53.739662 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:53.730184 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" podUID="3ad4d41c-7af3-453d-9005-d34264ed3acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 30 07:36:54.736600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:54.733025 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 30 07:36:54.736600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:54.733101 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" podUID="3ad4d41c-7af3-453d-9005-d34264ed3acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 30 07:36:55.738597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:55.738048 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 30 07:36:55.738597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:55.738130 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" podUID="3ad4d41c-7af3-453d-9005-d34264ed3acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 30 07:36:56.737062 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:56.737012 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 30 07:36:56.738317 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:56.738277 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" podUID="3ad4d41c-7af3-453d-9005-d34264ed3acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 30 07:36:57.744600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:57.742921 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 30 07:36:57.744600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:57.742982 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" podUID="3ad4d41c-7af3-453d-9005-d34264ed3acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 30 07:36:58.755940 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:58.755895 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:36:58.756609 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:58.756528 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:36:58.760624 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:36:58.760596 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 07:37:09.911412 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:09.858554 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb"] Oct 30 07:37:09.912473 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:09.912422 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" podUID="a398df37-12de-4a79-a5cb-d246f85cfa5f" containerName="istio-proxy" containerID="cri-o://670feba0db6aae9046b30f93d28b62fb77dfecf02e91fcf4a14926b073ec6395" gracePeriod=30 Oct 30 07:37:14.406336 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:14.406299 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t"] Oct 30 07:37:14.411884 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:14.411865 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t" Oct 30 07:37:14.450544 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:14.450510 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-a91a83-serving-cert\"" Oct 30 07:37:14.455758 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:14.455731 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t"] Oct 30 07:37:14.540332 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:14.540294 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abf16ce5-dabd-44dd-a441-9b7059c28338-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t\" (UID: \"abf16ce5-dabd-44dd-a441-9b7059c28338\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t" Oct 30 07:37:14.540515 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:14.540443 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf16ce5-dabd-44dd-a441-9b7059c28338-metrics-certs\") pod \"maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t\" (UID: \"abf16ce5-dabd-44dd-a441-9b7059c28338\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t" Oct 30 07:37:14.540560 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:14.540524 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf16ce5-dabd-44dd-a441-9b7059c28338-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t\" (UID: \"abf16ce5-dabd-44dd-a441-9b7059c28338\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t" Oct 30 07:37:14.540624 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:14.540601 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfpnq\" (UniqueName: \"kubernetes.io/projected/abf16ce5-dabd-44dd-a441-9b7059c28338-kube-api-access-gfpnq\") pod \"maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t\" (UID: \"abf16ce5-dabd-44dd-a441-9b7059c28338\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t" Oct 30 07:37:14.658641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:14.652692 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abf16ce5-dabd-44dd-a441-9b7059c28338-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t\" (UID: \"abf16ce5-dabd-44dd-a441-9b7059c28338\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t" Oct 30 07:37:14.658641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:14.652849 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf16ce5-dabd-44dd-a441-9b7059c28338-metrics-certs\") pod \"maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t\" (UID: \"abf16ce5-dabd-44dd-a441-9b7059c28338\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t" Oct 30 07:37:14.658641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:14.652888 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf16ce5-dabd-44dd-a441-9b7059c28338-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t\" (UID: \"abf16ce5-dabd-44dd-a441-9b7059c28338\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t" Oct 30 07:37:14.658641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:14.652921 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfpnq\" (UniqueName: \"kubernetes.io/projected/abf16ce5-dabd-44dd-a441-9b7059c28338-kube-api-access-gfpnq\") pod \"maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t\" (UID: \"abf16ce5-dabd-44dd-a441-9b7059c28338\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t" Oct 30 07:37:14.658641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:14.656351 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf16ce5-dabd-44dd-a441-9b7059c28338-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t\" (UID: \"abf16ce5-dabd-44dd-a441-9b7059c28338\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t" Oct 30 07:37:14.684782 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:14.684735 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf16ce5-dabd-44dd-a441-9b7059c28338-metrics-certs\") pod \"maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t\" (UID: \"abf16ce5-dabd-44dd-a441-9b7059c28338\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t" Oct 30 07:37:14.685692 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:14.685549 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abf16ce5-dabd-44dd-a441-9b7059c28338-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t\" (UID: \"abf16ce5-dabd-44dd-a441-9b7059c28338\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t" Oct 30 07:37:14.688054 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:14.688027 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfpnq\" (UniqueName: \"kubernetes.io/projected/abf16ce5-dabd-44dd-a441-9b7059c28338-kube-api-access-gfpnq\") pod \"maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t\" (UID: \"abf16ce5-dabd-44dd-a441-9b7059c28338\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t" Oct 30 07:37:14.727700 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:14.727672 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t" Oct 30 07:37:15.040154 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:37:15.040118 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf16ce5_dabd_44dd_a441_9b7059c28338.slice/crio-9f2c5660aead905de68650a37ce66b3e8cf973615d259291d7cf887bbd5f91ea WatchSource:0}: Error finding container 9f2c5660aead905de68650a37ce66b3e8cf973615d259291d7cf887bbd5f91ea: Status 404 returned error can't find the container with id 9f2c5660aead905de68650a37ce66b3e8cf973615d259291d7cf887bbd5f91ea Oct 30 07:37:15.040501 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.040475 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t"] Oct 30 07:37:15.041810 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.041790 4951 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 07:37:15.196180 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.196155 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:37:15.262644 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.262604 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-credential-socket\") pod \"a398df37-12de-4a79-a5cb-d246f85cfa5f\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " Oct 30 07:37:15.262891 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.262665 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a398df37-12de-4a79-a5cb-d246f85cfa5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a398df37-12de-4a79-a5cb-d246f85cfa5f-istio-podinfo\") pod \"a398df37-12de-4a79-a5cb-d246f85cfa5f\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " Oct 30 07:37:15.262891 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.262717 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-workload-socket\") pod \"a398df37-12de-4a79-a5cb-d246f85cfa5f\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " Oct 30 07:37:15.262891 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.262765 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-istio-data\") pod \"a398df37-12de-4a79-a5cb-d246f85cfa5f\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " Oct 30 07:37:15.262891 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.262861 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a398df37-12de-4a79-a5cb-d246f85cfa5f-istio-token\" (UniqueName: \"kubernetes.io/projected/a398df37-12de-4a79-a5cb-d246f85cfa5f-istio-token\") pod \"a398df37-12de-4a79-a5cb-d246f85cfa5f\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " Oct 30 07:37:15.263134 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.262896 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-workload-certs\") pod \"a398df37-12de-4a79-a5cb-d246f85cfa5f\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " Oct 30 07:37:15.263134 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.263034 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a398df37-12de-4a79-a5cb-d246f85cfa5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a398df37-12de-4a79-a5cb-d246f85cfa5f-istiod-ca-cert\") pod \"a398df37-12de-4a79-a5cb-d246f85cfa5f\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " Oct 30 07:37:15.263134 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.263063 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-istio-envoy\") pod \"a398df37-12de-4a79-a5cb-d246f85cfa5f\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " Oct 30 07:37:15.263328 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.263140 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a398df37-12de-4a79-a5cb-d246f85cfa5f-kube-api-access-pvwr8\" (UniqueName: \"kubernetes.io/projected/a398df37-12de-4a79-a5cb-d246f85cfa5f-kube-api-access-pvwr8\") pod \"a398df37-12de-4a79-a5cb-d246f85cfa5f\" (UID: \"a398df37-12de-4a79-a5cb-d246f85cfa5f\") " Oct 30 07:37:15.263801 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.263669 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-workload-certs" pod "a398df37-12de-4a79-a5cb-d246f85cfa5f" (UID: "a398df37-12de-4a79-a5cb-d246f85cfa5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:37:15.263801 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.263742 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-workload-socket" pod "a398df37-12de-4a79-a5cb-d246f85cfa5f" (UID: "a398df37-12de-4a79-a5cb-d246f85cfa5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:37:15.264023 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.263764 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a398df37-12de-4a79-a5cb-d246f85cfa5f-istiod-ca-cert" pod "a398df37-12de-4a79-a5cb-d246f85cfa5f" (UID: "a398df37-12de-4a79-a5cb-d246f85cfa5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:37:15.264023 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.263845 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-istio-data" pod "a398df37-12de-4a79-a5cb-d246f85cfa5f" (UID: "a398df37-12de-4a79-a5cb-d246f85cfa5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:37:15.264146 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.264044 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-credential-socket" pod "a398df37-12de-4a79-a5cb-d246f85cfa5f" (UID: "a398df37-12de-4a79-a5cb-d246f85cfa5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:37:15.264417 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.264397 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a398df37-12de-4a79-a5cb-d246f85cfa5f-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:37:15.264510 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.264422 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:37:15.264510 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.264436 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:37:15.264510 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.264451 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:37:15.264510 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.264465 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:37:15.266886 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.266854 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a398df37-12de-4a79-a5cb-d246f85cfa5f-kube-api-access-pvwr8" pod "a398df37-12de-4a79-a5cb-d246f85cfa5f" (UID: "a398df37-12de-4a79-a5cb-d246f85cfa5f"). InnerVolumeSpecName "kube-api-access-pvwr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:37:15.267008 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.266966 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a398df37-12de-4a79-a5cb-d246f85cfa5f-istio-token" pod "a398df37-12de-4a79-a5cb-d246f85cfa5f" (UID: "a398df37-12de-4a79-a5cb-d246f85cfa5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:37:15.267008 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.266971 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a398df37-12de-4a79-a5cb-d246f85cfa5f-istio-podinfo" pod "a398df37-12de-4a79-a5cb-d246f85cfa5f" (UID: "a398df37-12de-4a79-a5cb-d246f85cfa5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:37:15.267873 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.267850 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-istio-envoy" pod "a398df37-12de-4a79-a5cb-d246f85cfa5f" (UID: "a398df37-12de-4a79-a5cb-d246f85cfa5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:37:15.365864 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.365827 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a398df37-12de-4a79-a5cb-d246f85cfa5f-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:37:15.365864 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.365866 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a398df37-12de-4a79-a5cb-d246f85cfa5f-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:37:15.366090 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.365879 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a398df37-12de-4a79-a5cb-d246f85cfa5f-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:37:15.366090 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.365893 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvwr8\" (UniqueName: \"kubernetes.io/projected/a398df37-12de-4a79-a5cb-d246f85cfa5f-kube-api-access-pvwr8\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:37:15.520633 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.520597 4951 generic.go:356] "Generic (PLEG): container finished" podID="a398df37-12de-4a79-a5cb-d246f85cfa5f" containerID="670feba0db6aae9046b30f93d28b62fb77dfecf02e91fcf4a14926b073ec6395" exitCode=0 Oct 30 07:37:15.521237 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.520651 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" Oct 30 07:37:15.521237 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.520684 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" event={"ID":"a398df37-12de-4a79-a5cb-d246f85cfa5f","Type":"ContainerDied","Data":"670feba0db6aae9046b30f93d28b62fb77dfecf02e91fcf4a14926b073ec6395"} Oct 30 07:37:15.521237 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.520737 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb" event={"ID":"a398df37-12de-4a79-a5cb-d246f85cfa5f","Type":"ContainerDied","Data":"5635791462f1938ce1029de2c2dd84cacae9ebec1bd3357d33849009ad8fe054"} Oct 30 07:37:15.521237 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.520768 4951 scope.go:122] "RemoveContainer" containerID="670feba0db6aae9046b30f93d28b62fb77dfecf02e91fcf4a14926b073ec6395" Oct 30 07:37:15.522165 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.522126 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t" event={"ID":"abf16ce5-dabd-44dd-a441-9b7059c28338","Type":"ContainerStarted","Data":"9f2c5660aead905de68650a37ce66b3e8cf973615d259291d7cf887bbd5f91ea"} Oct 30 07:37:15.533418 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.533400 4951 scope.go:122] "RemoveContainer" containerID="670feba0db6aae9046b30f93d28b62fb77dfecf02e91fcf4a14926b073ec6395" Oct 30 07:37:15.533858 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:37:15.533839 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"670feba0db6aae9046b30f93d28b62fb77dfecf02e91fcf4a14926b073ec6395\": container with ID starting with 670feba0db6aae9046b30f93d28b62fb77dfecf02e91fcf4a14926b073ec6395 not found: ID does not exist" containerID="670feba0db6aae9046b30f93d28b62fb77dfecf02e91fcf4a14926b073ec6395" Oct 30 07:37:15.533937 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.533864 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"670feba0db6aae9046b30f93d28b62fb77dfecf02e91fcf4a14926b073ec6395"} err="failed to get container status \"670feba0db6aae9046b30f93d28b62fb77dfecf02e91fcf4a14926b073ec6395\": rpc error: code = NotFound desc = could not find container \"670feba0db6aae9046b30f93d28b62fb77dfecf02e91fcf4a14926b073ec6395\": container with ID starting with 670feba0db6aae9046b30f93d28b62fb77dfecf02e91fcf4a14926b073ec6395 not found: ID does not exist" Oct 30 07:37:15.582400 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.582374 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb"] Oct 30 07:37:15.589213 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.589190 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-1fb038-openshift-default-6cdb46fb9-hmdqb"] Oct 30 07:37:15.774255 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:15.774178 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a398df37-12de-4a79-a5cb-d246f85cfa5f" path="/var/lib/kubelet/pods/a398df37-12de-4a79-a5cb-d246f85cfa5f/volumes" Oct 30 07:37:16.532505 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:16.532464 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t" event={"ID":"abf16ce5-dabd-44dd-a441-9b7059c28338","Type":"ContainerStarted","Data":"62d745432c0b6accb3395712c7b5ef79480a64a5a87d0847b602c591c8e640ab"} Oct 30 07:37:16.532962 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:16.532708 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t" Oct 30 07:37:16.582917 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:16.582850 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t" podStartSLOduration=2.062403278 podStartE2EDuration="2.582834498s" totalImagesPullingTime="520.43122ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:37:16.579123405 +0000 UTC m=+2442.922171391" watchObservedRunningTime="2026-10-30 07:37:16.582834498 +0000 UTC m=+2442.925882463" Oct 30 07:37:19.602935 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:19.602894 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw"] Oct 30 07:37:19.603655 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:19.603636 4951 cpu_manager.go:414] "containerMap: removing container" podUID="a398df37-12de-4a79-a5cb-d246f85cfa5f" containerName="istio-proxy" Oct 30 07:37:19.603723 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:19.603657 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a398df37-12de-4a79-a5cb-d246f85cfa5f" containerName="istio-proxy" Oct 30 07:37:19.603826 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:19.603816 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="a398df37-12de-4a79-a5cb-d246f85cfa5f" containerName="istio-proxy" Oct 30 07:37:19.608860 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:19.608832 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw" Oct 30 07:37:19.642042 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:19.641089 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-bffe43-serving-cert\"" Oct 30 07:37:19.646745 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:19.646521 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw"] Oct 30 07:37:19.678098 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:19.678065 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffb358d7-f1b3-4e53-b19e-85031c692e2b-metrics-certs\") pod \"maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw\" (UID: \"ffb358d7-f1b3-4e53-b19e-85031c692e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw" Oct 30 07:37:19.678269 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:19.678217 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffb358d7-f1b3-4e53-b19e-85031c692e2b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw\" (UID: \"ffb358d7-f1b3-4e53-b19e-85031c692e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw" Oct 30 07:37:19.678269 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:19.678257 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffb358d7-f1b3-4e53-b19e-85031c692e2b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw\" (UID: \"ffb358d7-f1b3-4e53-b19e-85031c692e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw" Oct 30 07:37:19.678369 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:19.678358 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmccp\" (UniqueName: \"kubernetes.io/projected/ffb358d7-f1b3-4e53-b19e-85031c692e2b-kube-api-access-bmccp\") pod \"maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw\" (UID: \"ffb358d7-f1b3-4e53-b19e-85031c692e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw" Oct 30 07:37:19.779333 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:19.779293 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffb358d7-f1b3-4e53-b19e-85031c692e2b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw\" (UID: \"ffb358d7-f1b3-4e53-b19e-85031c692e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw" Oct 30 07:37:19.779551 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:19.779362 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffb358d7-f1b3-4e53-b19e-85031c692e2b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw\" (UID: \"ffb358d7-f1b3-4e53-b19e-85031c692e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw" Oct 30 07:37:19.779551 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:19.779458 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmccp\" (UniqueName: \"kubernetes.io/projected/ffb358d7-f1b3-4e53-b19e-85031c692e2b-kube-api-access-bmccp\") pod \"maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw\" (UID: \"ffb358d7-f1b3-4e53-b19e-85031c692e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw" Oct 30 07:37:19.779716 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:19.779657 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffb358d7-f1b3-4e53-b19e-85031c692e2b-metrics-certs\") pod \"maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw\" (UID: \"ffb358d7-f1b3-4e53-b19e-85031c692e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw" Oct 30 07:37:19.780085 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:19.780062 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffb358d7-f1b3-4e53-b19e-85031c692e2b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw\" (UID: \"ffb358d7-f1b3-4e53-b19e-85031c692e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw" Oct 30 07:37:19.782088 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:19.782058 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffb358d7-f1b3-4e53-b19e-85031c692e2b-metrics-certs\") pod \"maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw\" (UID: \"ffb358d7-f1b3-4e53-b19e-85031c692e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw" Oct 30 07:37:19.782192 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:19.782105 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffb358d7-f1b3-4e53-b19e-85031c692e2b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw\" (UID: \"ffb358d7-f1b3-4e53-b19e-85031c692e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw" Oct 30 07:37:19.804437 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:19.804414 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmccp\" (UniqueName: \"kubernetes.io/projected/ffb358d7-f1b3-4e53-b19e-85031c692e2b-kube-api-access-bmccp\") pod \"maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw\" (UID: \"ffb358d7-f1b3-4e53-b19e-85031c692e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw" Oct 30 07:37:19.933240 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:19.932877 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw" Oct 30 07:37:20.256837 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:37:20.256801 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffb358d7_f1b3_4e53_b19e_85031c692e2b.slice/crio-7c65dabb961feac6d0f5b6081dd74216333cd76faca9cc430f8658920ab5cf6d WatchSource:0}: Error finding container 7c65dabb961feac6d0f5b6081dd74216333cd76faca9cc430f8658920ab5cf6d: Status 404 returned error can't find the container with id 7c65dabb961feac6d0f5b6081dd74216333cd76faca9cc430f8658920ab5cf6d Oct 30 07:37:20.257001 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:20.256896 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw"] Oct 30 07:37:20.566540 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:20.566457 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw" event={"ID":"ffb358d7-f1b3-4e53-b19e-85031c692e2b","Type":"ContainerStarted","Data":"7c65dabb961feac6d0f5b6081dd74216333cd76faca9cc430f8658920ab5cf6d"} Oct 30 07:37:21.594664 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:21.594172 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw" event={"ID":"ffb358d7-f1b3-4e53-b19e-85031c692e2b","Type":"ContainerStarted","Data":"187fb754eecfd756b51be689da6df481cbc4746747d79bcb10115e3cf14a3377"} Oct 30 07:37:21.596468 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:21.595427 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw" Oct 30 07:37:21.649623 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:21.649368 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw" podStartSLOduration=2.295359413 podStartE2EDuration="2.649352768s" totalImagesPullingTime="353.993355ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:37:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:37:21.64433131 +0000 UTC m=+2447.987379276" watchObservedRunningTime="2026-10-30 07:37:21.649352768 +0000 UTC m=+2447.992400732" Oct 30 07:37:22.547982 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:22.547949 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t" Oct 30 07:37:23.024107 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:37:23.024074 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:37:28.613891 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:28.613847 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw" Oct 30 07:37:38.317369 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:38.315906 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": read tcp 10.132.0.2:33874->10.132.0.225:15021: read: connection reset by peer" start-of-body= Oct 30 07:37:38.317369 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:38.315968 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" podUID="d0c9697a-7dea-4662-8da4-c2618945d160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": read tcp 10.132.0.2:33874->10.132.0.225:15021: read: connection reset by peer" Oct 30 07:37:38.730465 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:38.730376 4951 generic.go:356] "Generic (PLEG): container finished" podID="d0c9697a-7dea-4662-8da4-c2618945d160" containerID="4c0dcfe40ef7c95e5594b253f38058adc8af6f1c0156d183282f1d07aa052521" exitCode=0 Oct 30 07:37:38.730651 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:38.730461 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" event={"ID":"d0c9697a-7dea-4662-8da4-c2618945d160","Type":"ContainerDied","Data":"4c0dcfe40ef7c95e5594b253f38058adc8af6f1c0156d183282f1d07aa052521"} Oct 30 07:37:38.731107 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:38.731081 4951 scope.go:122] "RemoveContainer" containerID="4c0dcfe40ef7c95e5594b253f38058adc8af6f1c0156d183282f1d07aa052521" Oct 30 07:37:38.734208 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:38.734168 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:37:38.734295 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:38.734270 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:37:38.734344 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:38.734319 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:37:39.762184 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:39.762150 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" event={"ID":"d0c9697a-7dea-4662-8da4-c2618945d160","Type":"ContainerStarted","Data":"96f82bd1c8cc328ddf780900ba3326e9bf8394f6ae1ad18a83f832f3b8238e90"} Oct 30 07:37:40.176366 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:40.176273 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:37:40.176555 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:40.176481 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 07:37:40.176555 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:40.176525 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" podUID="d0c9697a-7dea-4662-8da4-c2618945d160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 07:37:41.175958 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:41.175921 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 07:37:41.176354 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:41.175976 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" podUID="d0c9697a-7dea-4662-8da4-c2618945d160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 07:37:41.749754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:41.749712 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4"] Oct 30 07:37:41.750042 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:41.750014 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" podUID="bd13b861-9db1-4e94-bade-fa1a2ca4b04d" containerName="istio-proxy" containerID="cri-o://7248cda4ade74431263bfdaccff9879761c779383ca885e91c8e80e391c31c7f" gracePeriod=30 Oct 30 07:37:42.176162 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:42.176068 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 07:37:42.176729 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:42.176690 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" podUID="d0c9697a-7dea-4662-8da4-c2618945d160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 07:37:43.176468 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:43.176435 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 07:37:43.177012 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:43.176484 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" podUID="d0c9697a-7dea-4662-8da4-c2618945d160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 07:37:44.176913 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:44.176872 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 07:37:44.177360 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:44.176944 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" podUID="d0c9697a-7dea-4662-8da4-c2618945d160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 07:37:45.176679 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:45.176647 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 30 07:37:45.176908 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:45.176706 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" podUID="d0c9697a-7dea-4662-8da4-c2618945d160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 30 07:37:46.181621 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:46.181530 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:37:46.182039 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:46.182014 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:37:46.182737 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:46.182718 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 07:37:46.597513 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:46.597481 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst"] Oct 30 07:37:46.604875 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:46.604844 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst" Oct 30 07:37:46.623999 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:46.623967 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-0f76b0-serving-cert\"" Oct 30 07:37:46.631109 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:46.631059 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst"] Oct 30 07:37:46.730762 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:46.730723 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2149f8b0-9707-4e35-851e-af05b7842bf3-metrics-certs\") pod \"maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst\" (UID: \"2149f8b0-9707-4e35-851e-af05b7842bf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst" Oct 30 07:37:46.731094 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:46.731067 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2149f8b0-9707-4e35-851e-af05b7842bf3-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst\" (UID: \"2149f8b0-9707-4e35-851e-af05b7842bf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst" Oct 30 07:37:46.731171 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:46.731130 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2149f8b0-9707-4e35-851e-af05b7842bf3-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst\" (UID: \"2149f8b0-9707-4e35-851e-af05b7842bf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst" Oct 30 07:37:46.731298 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:46.731269 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfkln\" (UniqueName: \"kubernetes.io/projected/2149f8b0-9707-4e35-851e-af05b7842bf3-kube-api-access-nfkln\") pod \"maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst\" (UID: \"2149f8b0-9707-4e35-851e-af05b7842bf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst" Oct 30 07:37:46.836015 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:46.833635 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfkln\" (UniqueName: \"kubernetes.io/projected/2149f8b0-9707-4e35-851e-af05b7842bf3-kube-api-access-nfkln\") pod \"maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst\" (UID: \"2149f8b0-9707-4e35-851e-af05b7842bf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst" Oct 30 07:37:46.836015 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:46.833736 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2149f8b0-9707-4e35-851e-af05b7842bf3-metrics-certs\") pod \"maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst\" (UID: \"2149f8b0-9707-4e35-851e-af05b7842bf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst" Oct 30 07:37:46.836015 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:46.834017 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2149f8b0-9707-4e35-851e-af05b7842bf3-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst\" (UID: \"2149f8b0-9707-4e35-851e-af05b7842bf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst" Oct 30 07:37:46.836015 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:46.834068 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2149f8b0-9707-4e35-851e-af05b7842bf3-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst\" (UID: \"2149f8b0-9707-4e35-851e-af05b7842bf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst" Oct 30 07:37:46.836015 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:46.834992 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2149f8b0-9707-4e35-851e-af05b7842bf3-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst\" (UID: \"2149f8b0-9707-4e35-851e-af05b7842bf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst" Oct 30 07:37:46.836450 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:37:46.836321 4951 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f76b0-serving-cert: secret "maas-api-e2e-extmod-b-0f76b0-serving-cert" not found Oct 30 07:37:46.836450 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:37:46.836401 4951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2149f8b0-9707-4e35-851e-af05b7842bf3-maas-api-tls podName:2149f8b0-9707-4e35-851e-af05b7842bf3 nodeName:}" failed. No retries permitted until 2026-10-30 07:37:47.336378104 +0000 UTC m=+2473.679426053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2149f8b0-9707-4e35-851e-af05b7842bf3-maas-api-tls") pod "maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst" (UID: "2149f8b0-9707-4e35-851e-af05b7842bf3") : secret "maas-api-e2e-extmod-b-0f76b0-serving-cert" not found Oct 30 07:37:46.844928 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:46.844895 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2149f8b0-9707-4e35-851e-af05b7842bf3-metrics-certs\") pod \"maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst\" (UID: \"2149f8b0-9707-4e35-851e-af05b7842bf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst" Oct 30 07:37:46.861297 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:46.861222 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfkln\" (UniqueName: \"kubernetes.io/projected/2149f8b0-9707-4e35-851e-af05b7842bf3-kube-api-access-nfkln\") pod \"maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst\" (UID: \"2149f8b0-9707-4e35-851e-af05b7842bf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst" Oct 30 07:37:46.884515 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:46.884469 4951 generic.go:356] "Generic (PLEG): container finished" podID="bd13b861-9db1-4e94-bade-fa1a2ca4b04d" containerID="7248cda4ade74431263bfdaccff9879761c779383ca885e91c8e80e391c31c7f" exitCode=0 Oct 30 07:37:46.884741 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:46.884533 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" event={"ID":"bd13b861-9db1-4e94-bade-fa1a2ca4b04d","Type":"ContainerDied","Data":"7248cda4ade74431263bfdaccff9879761c779383ca885e91c8e80e391c31c7f"} Oct 30 07:37:47.053064 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.053029 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:37:47.137820 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.137730 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istio-data\") pod \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " Oct 30 07:37:47.137995 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.137836 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istiod-ca-cert\") pod \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " Oct 30 07:37:47.137995 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.137862 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istio-token\" (UniqueName: \"kubernetes.io/projected/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istio-token\") pod \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " Oct 30 07:37:47.137995 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.137968 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-workload-socket\") pod \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " Oct 30 07:37:47.138168 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.138037 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-credential-socket\") pod \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " Oct 30 07:37:47.138168 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.138065 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-kube-api-access-6kj98\" (UniqueName: \"kubernetes.io/projected/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-kube-api-access-6kj98\") pod \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " Oct 30 07:37:47.138168 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.138089 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istio-envoy\") pod \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " Oct 30 07:37:47.138168 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.138117 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-workload-certs\") pod \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " Oct 30 07:37:47.138385 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.138175 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istio-podinfo\") pod \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\" (UID: \"bd13b861-9db1-4e94-bade-fa1a2ca4b04d\") " Oct 30 07:37:47.138451 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.138412 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istio-data" pod "bd13b861-9db1-4e94-bade-fa1a2ca4b04d" (UID: "bd13b861-9db1-4e94-bade-fa1a2ca4b04d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:37:47.138510 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.138424 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-credential-socket" pod "bd13b861-9db1-4e94-bade-fa1a2ca4b04d" (UID: "bd13b861-9db1-4e94-bade-fa1a2ca4b04d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:37:47.138510 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.138489 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-workload-socket" pod "bd13b861-9db1-4e94-bade-fa1a2ca4b04d" (UID: "bd13b861-9db1-4e94-bade-fa1a2ca4b04d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:37:47.138736 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.138712 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-workload-certs" pod "bd13b861-9db1-4e94-bade-fa1a2ca4b04d" (UID: "bd13b861-9db1-4e94-bade-fa1a2ca4b04d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:37:47.138900 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.138879 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:37:47.138983 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.138901 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:37:47.138983 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.138910 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:37:47.138983 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.138924 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:37:47.138983 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.138920 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istiod-ca-cert" pod "bd13b861-9db1-4e94-bade-fa1a2ca4b04d" (UID: "bd13b861-9db1-4e94-bade-fa1a2ca4b04d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:37:47.140776 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.140749 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istio-podinfo" pod "bd13b861-9db1-4e94-bade-fa1a2ca4b04d" (UID: "bd13b861-9db1-4e94-bade-fa1a2ca4b04d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:37:47.141220 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.141201 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istio-token" pod "bd13b861-9db1-4e94-bade-fa1a2ca4b04d" (UID: "bd13b861-9db1-4e94-bade-fa1a2ca4b04d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:37:47.141287 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.141229 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-kube-api-access-6kj98" pod "bd13b861-9db1-4e94-bade-fa1a2ca4b04d" (UID: "bd13b861-9db1-4e94-bade-fa1a2ca4b04d"). InnerVolumeSpecName "kube-api-access-6kj98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:37:47.141367 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.141351 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istio-envoy" pod "bd13b861-9db1-4e94-bade-fa1a2ca4b04d" (UID: "bd13b861-9db1-4e94-bade-fa1a2ca4b04d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:37:47.240360 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.240319 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:37:47.240360 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.240354 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:37:47.240838 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.240368 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kj98\" (UniqueName: \"kubernetes.io/projected/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-kube-api-access-6kj98\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:37:47.240838 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.240383 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:37:47.240838 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.240395 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd13b861-9db1-4e94-bade-fa1a2ca4b04d-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:37:47.341730 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.341698 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2149f8b0-9707-4e35-851e-af05b7842bf3-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst\" (UID: \"2149f8b0-9707-4e35-851e-af05b7842bf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst" Oct 30 07:37:47.344352 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.344313 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2149f8b0-9707-4e35-851e-af05b7842bf3-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst\" (UID: \"2149f8b0-9707-4e35-851e-af05b7842bf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst" Oct 30 07:37:47.534705 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.534659 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst" Oct 30 07:37:47.844290 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:37:47.844258 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2149f8b0_9707_4e35_851e_af05b7842bf3.slice/crio-49650bfff22b4994fbf9d477cd4172a00d19f3fb331fe8ed522fdf4d18861b9a WatchSource:0}: Error finding container 49650bfff22b4994fbf9d477cd4172a00d19f3fb331fe8ed522fdf4d18861b9a: Status 404 returned error can't find the container with id 49650bfff22b4994fbf9d477cd4172a00d19f3fb331fe8ed522fdf4d18861b9a Oct 30 07:37:47.844669 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.844563 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst"] Oct 30 07:37:47.936591 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.933183 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" event={"ID":"bd13b861-9db1-4e94-bade-fa1a2ca4b04d","Type":"ContainerDied","Data":"170b624da3cf93e62f85c6f4bbd1bb5f4ffaf42a0d5475e94335bd5e13440685"} Oct 30 07:37:47.936591 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.933265 4951 scope.go:122] "RemoveContainer" containerID="7248cda4ade74431263bfdaccff9879761c779383ca885e91c8e80e391c31c7f" Oct 30 07:37:47.936591 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.933428 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4" Oct 30 07:37:47.947804 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.943503 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst" event={"ID":"2149f8b0-9707-4e35-851e-af05b7842bf3","Type":"ContainerStarted","Data":"49650bfff22b4994fbf9d477cd4172a00d19f3fb331fe8ed522fdf4d18861b9a"} Oct 30 07:37:47.997438 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:47.997395 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4"] Oct 30 07:37:48.001684 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:48.001633 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-mhgn4"] Oct 30 07:37:48.952509 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:48.952470 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst" event={"ID":"2149f8b0-9707-4e35-851e-af05b7842bf3","Type":"ContainerStarted","Data":"d68cb0c0eb70fc03f9f95b0287047f21cb273108e7294f9a6709ef486cddb9c1"} Oct 30 07:37:48.953004 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:48.952547 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst" Oct 30 07:37:49.002179 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:49.002121 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst" podStartSLOduration=2.561945853 podStartE2EDuration="3.002103863s" totalImagesPullingTime="440.15801ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:37:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:37:48.997775467 +0000 UTC m=+2475.340823423" watchObservedRunningTime="2026-10-30 07:37:49.002103863 +0000 UTC m=+2475.345151830" Oct 30 07:37:49.772715 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:49.772681 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd13b861-9db1-4e94-bade-fa1a2ca4b04d" path="/var/lib/kubelet/pods/bd13b861-9db1-4e94-bade-fa1a2ca4b04d/volumes" Oct 30 07:37:53.027387 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:37:53.027343 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:37:54.968095 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:37:54.968064 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst" Oct 30 07:38:03.227828 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.227785 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz"] Oct 30 07:38:03.229433 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.229391 4951 cpu_manager.go:414] "containerMap: removing container" podUID="bd13b861-9db1-4e94-bade-fa1a2ca4b04d" containerName="istio-proxy" Oct 30 07:38:03.229433 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.229429 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd13b861-9db1-4e94-bade-fa1a2ca4b04d" containerName="istio-proxy" Oct 30 07:38:03.229908 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.229659 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd13b861-9db1-4e94-bade-fa1a2ca4b04d" containerName="istio-proxy" Oct 30 07:38:03.233822 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.233800 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:38:03.264858 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.261136 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz"] Oct 30 07:38:03.265062 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.265014 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-b3a7d7-openshift-default-dockercfg-qr7nm\"" Oct 30 07:38:03.395721 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.395682 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-credential-socket\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:38:03.395927 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.395730 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-workload-socket\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:38:03.395927 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.395810 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-istio-data\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:38:03.395927 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.395908 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-workload-certs\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:38:03.396106 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.395940 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1304cf8-fc22-48db-acce-17514c25cf9d-istio-podinfo\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:38:03.396106 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.396050 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1304cf8-fc22-48db-acce-17514c25cf9d-istio-token\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:38:03.396189 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.396101 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1304cf8-fc22-48db-acce-17514c25cf9d-istiod-ca-cert\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:38:03.396258 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.396240 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpjg8\" (UniqueName: \"kubernetes.io/projected/e1304cf8-fc22-48db-acce-17514c25cf9d-kube-api-access-dpjg8\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:38:03.396295 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.396274 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-istio-envoy\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:38:03.502685 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.499756 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-credential-socket\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:38:03.502685 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.499817 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-workload-socket\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:38:03.502685 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.499850 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-istio-data\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:38:03.502685 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.499952 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-workload-certs\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:38:03.502685 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.499994 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1304cf8-fc22-48db-acce-17514c25cf9d-istio-podinfo\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:38:03.502685 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.500035 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1304cf8-fc22-48db-acce-17514c25cf9d-istio-token\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:38:03.502685 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.500073 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1304cf8-fc22-48db-acce-17514c25cf9d-istiod-ca-cert\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:38:03.502685 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.500191 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpjg8\" (UniqueName: \"kubernetes.io/projected/e1304cf8-fc22-48db-acce-17514c25cf9d-kube-api-access-dpjg8\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:38:03.502685 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.500226 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-istio-envoy\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:38:03.530611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.527109 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-workload-certs\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:38:03.530611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.527398 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-credential-socket\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:38:03.530611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.527681 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-workload-socket\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:38:03.530611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.528003 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-istio-data\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:38:03.536605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.531791 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1304cf8-fc22-48db-acce-17514c25cf9d-istiod-ca-cert\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:38:03.547610 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.544238 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-istio-envoy\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:38:03.547610 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.546117 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1304cf8-fc22-48db-acce-17514c25cf9d-istio-podinfo\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:38:03.575559 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.571381 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpjg8\" (UniqueName: \"kubernetes.io/projected/e1304cf8-fc22-48db-acce-17514c25cf9d-kube-api-access-dpjg8\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:38:03.586595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.584215 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1304cf8-fc22-48db-acce-17514c25cf9d-istio-token\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:38:03.849486 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:03.849309 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:38:04.196548 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:04.196519 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz"] Oct 30 07:38:04.199751 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:04.199709 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:38:04.199848 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:04.199796 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:38:04.199897 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:04.199845 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:38:05.091506 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:05.091470 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" event={"ID":"e1304cf8-fc22-48db-acce-17514c25cf9d","Type":"ContainerStarted","Data":"36d54d722f8f8b476a866fbab288a0b6c294c4bc6363a856345867f3360517d4"} Oct 30 07:38:05.091996 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:05.091537 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" event={"ID":"e1304cf8-fc22-48db-acce-17514c25cf9d","Type":"ContainerStarted","Data":"6a5cfdb780d74637336a72babbee10f63831b571ba6eef72f3f9e4638b61839b"} Oct 30 07:38:05.188897 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:05.188833 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" podStartSLOduration=2.188814941 podStartE2EDuration="2.188814941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:38:05.185423111 +0000 UTC m=+2491.528471079" watchObservedRunningTime="2026-10-30 07:38:05.188814941 +0000 UTC m=+2491.531862911" Oct 30 07:38:05.852978 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:05.852937 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:38:05.860055 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:05.860027 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:38:06.114832 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:06.114666 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:38:06.117743 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:06.117336 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:38:12.201767 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:12.201725 4951 patch_prober.go:29] interesting pod/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:38:12.202223 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:12.201789 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" podUID="f52156d3-44fc-4983-984c-8ac36b1cb942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:38:14.246867 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:14.246777 4951 generic.go:356] "Generic (PLEG): container finished" podID="f52156d3-44fc-4983-984c-8ac36b1cb942" containerID="0d56edeb5190f4570765bd8943b7ca7c92294d37c62e21b5831dcb036331a833" exitCode=0 Oct 30 07:38:14.246867 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:14.246851 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" event={"ID":"f52156d3-44fc-4983-984c-8ac36b1cb942","Type":"ContainerDied","Data":"0d56edeb5190f4570765bd8943b7ca7c92294d37c62e21b5831dcb036331a833"} Oct 30 07:38:14.247413 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:14.247354 4951 scope.go:122] "RemoveContainer" containerID="0d56edeb5190f4570765bd8943b7ca7c92294d37c62e21b5831dcb036331a833" Oct 30 07:38:14.249629 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:14.249591 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:38:14.249734 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:14.249688 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:38:14.249775 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:14.249734 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:38:15.263940 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:15.263896 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" event={"ID":"f52156d3-44fc-4983-984c-8ac36b1cb942","Type":"ContainerStarted","Data":"221f7866e13f9415e02a182c85468354be59fe340d67e59e135aa0d0746a5417"} Oct 30 07:38:16.210655 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:16.210614 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:38:16.210883 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:16.210819 4951 patch_prober.go:29] interesting pod/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 07:38:16.210941 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:16.210876 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" podUID="f52156d3-44fc-4983-984c-8ac36b1cb942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 07:38:17.210366 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:17.210319 4951 patch_prober.go:29] interesting pod/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 07:38:17.210824 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:17.210406 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" podUID="f52156d3-44fc-4983-984c-8ac36b1cb942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 07:38:18.209867 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:18.209822 4951 patch_prober.go:29] interesting pod/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 07:38:18.210063 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:18.209880 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" podUID="f52156d3-44fc-4983-984c-8ac36b1cb942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 07:38:19.210150 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:19.210118 4951 patch_prober.go:29] interesting pod/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 07:38:19.210747 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:19.210708 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" podUID="f52156d3-44fc-4983-984c-8ac36b1cb942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 07:38:20.211369 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:20.211334 4951 patch_prober.go:29] interesting pod/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 07:38:20.211914 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:20.211394 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" podUID="f52156d3-44fc-4983-984c-8ac36b1cb942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 07:38:21.214714 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:21.214678 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:38:21.215158 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:21.215138 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:38:21.215841 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:38:21.215820 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 07:38:23.026420 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:38:23.026383 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:38:53.025631 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:38:53.025595 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:39:23.142833 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:39:23.142739 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:39:26.817677 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:26.817635 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc"] Oct 30 07:39:26.862369 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:26.862337 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc"] Oct 30 07:39:26.862668 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:26.862480 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:39:26.884969 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:26.884938 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-bb2e8f-openshift-default-dockercfg-b7ctc\"" Oct 30 07:39:26.986595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:26.952185 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-credential-socket\") pod \"e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:39:26.986958 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:26.986927 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istio-podinfo\") pod \"e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:39:26.987238 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:26.987214 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istio-envoy\") pod \"e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:39:27.047503 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:27.047465 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-workload-socket\") pod \"e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:39:27.047896 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:27.047871 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9qhp\" (UniqueName: \"kubernetes.io/projected/b08b669b-c0a7-462c-9b63-aabbb0254dc1-kube-api-access-n9qhp\") pod \"e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:39:27.048217 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:27.048191 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-workload-certs\") pod \"e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:39:27.048423 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:27.048406 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istio-token\") pod \"e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:39:27.048642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:27.048623 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istio-data\") pod \"e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:39:27.048821 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:27.048802 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istiod-ca-cert\") pod \"e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:39:27.155439 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:27.155342 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-credential-socket\") pod \"e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:39:27.155760 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:27.155732 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istio-podinfo\") pod \"e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:39:27.155961 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:27.155942 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istio-envoy\") pod \"e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:39:27.156115 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:27.156100 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-workload-socket\") pod \"e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:39:27.156262 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:27.156246 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9qhp\" (UniqueName: \"kubernetes.io/projected/b08b669b-c0a7-462c-9b63-aabbb0254dc1-kube-api-access-n9qhp\") pod \"e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:39:27.156430 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:27.156415 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-workload-certs\") pod \"e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:39:27.156549 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:27.156533 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istio-token\") pod \"e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:39:27.156816 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:27.156706 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istio-data\") pod \"e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:39:27.156952 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:27.156936 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istiod-ca-cert\") pod \"e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:39:27.158039 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:27.157991 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istiod-ca-cert\") pod \"e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:39:27.159465 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:27.158308 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-workload-socket\") pod \"e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:39:27.159635 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:27.158519 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-credential-socket\") pod \"e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:39:27.160491 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:27.160457 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-workload-certs\") pod \"e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:39:27.161536 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:27.161516 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istio-data\") pod \"e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:39:27.189221 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:27.189088 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istio-podinfo\") pod \"e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:39:27.190307 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:27.190206 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istio-envoy\") pod \"e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:39:27.191389 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:27.190905 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istio-token\") pod \"e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:39:27.191640 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:27.191594 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9qhp\" (UniqueName: \"kubernetes.io/projected/b08b669b-c0a7-462c-9b63-aabbb0254dc1-kube-api-access-n9qhp\") pod \"e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:39:27.490952 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:27.490860 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:39:27.855492 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:27.855466 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc"] Oct 30 07:39:27.856212 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:39:27.856186 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb08b669b_c0a7_462c_9b63_aabbb0254dc1.slice/crio-25d313a1081d63874cb163f1f3167e7554db37f47a98dd5e0b0bd4336213f08b WatchSource:0}: Error finding container 25d313a1081d63874cb163f1f3167e7554db37f47a98dd5e0b0bd4336213f08b: Status 404 returned error can't find the container with id 25d313a1081d63874cb163f1f3167e7554db37f47a98dd5e0b0bd4336213f08b Oct 30 07:39:27.859186 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:27.859150 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:39:27.859272 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:27.859216 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:39:27.859272 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:27.859245 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:39:27.916354 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:27.916322 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" event={"ID":"b08b669b-c0a7-462c-9b63-aabbb0254dc1","Type":"ContainerStarted","Data":"25d313a1081d63874cb163f1f3167e7554db37f47a98dd5e0b0bd4336213f08b"} Oct 30 07:39:28.957036 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:28.956983 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" event={"ID":"b08b669b-c0a7-462c-9b63-aabbb0254dc1","Type":"ContainerStarted","Data":"dfe77b3f6c8dbc03e45aaa35620dc0846e8baba105df5275450257231527b35d"} Oct 30 07:39:29.041597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:29.041403 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" podStartSLOduration=3.041376972 podStartE2EDuration="3.041376972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:39:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:39:29.01883609 +0000 UTC m=+2575.361884056" watchObservedRunningTime="2026-10-30 07:39:29.041376972 +0000 UTC m=+2575.384424935" Oct 30 07:39:29.491993 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:29.491959 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:39:29.499244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:29.499216 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:39:29.967607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:29.967514 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:39:29.968607 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:29.968554 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:39:34.000881 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:34.000848 4951 generic.go:356] "Generic (PLEG): container finished" podID="e1304cf8-fc22-48db-acce-17514c25cf9d" containerID="36d54d722f8f8b476a866fbab288a0b6c294c4bc6363a856345867f3360517d4" exitCode=0 Oct 30 07:39:34.001372 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:34.000922 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" event={"ID":"e1304cf8-fc22-48db-acce-17514c25cf9d","Type":"ContainerDied","Data":"36d54d722f8f8b476a866fbab288a0b6c294c4bc6363a856345867f3360517d4"} Oct 30 07:39:34.001509 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:34.001492 4951 scope.go:122] "RemoveContainer" containerID="36d54d722f8f8b476a866fbab288a0b6c294c4bc6363a856345867f3360517d4" Oct 30 07:39:34.003610 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:34.003563 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:39:34.003749 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:34.003673 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:39:34.003749 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:34.003728 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:39:35.011299 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:35.011268 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" event={"ID":"e1304cf8-fc22-48db-acce-17514c25cf9d","Type":"ContainerStarted","Data":"0da3881c82a73b08b3e9def617785842b367756f987dcf7bce4100d3ea5b6d31"} Oct 30 07:39:35.858853 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:35.858817 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:39:35.860614 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:35.860416 4951 patch_prober.go:29] interesting pod/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 07:39:35.860614 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:35.860477 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" podUID="e1304cf8-fc22-48db-acce-17514c25cf9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 07:39:36.111275 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:36.111185 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:39:36.850023 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:36.849981 4951 patch_prober.go:29] interesting pod/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 07:39:36.850323 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:36.850292 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" podUID="e1304cf8-fc22-48db-acce-17514c25cf9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 07:39:37.855177 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:37.855142 4951 patch_prober.go:29] interesting pod/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 07:39:37.858133 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:37.858088 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" podUID="e1304cf8-fc22-48db-acce-17514c25cf9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 07:39:38.864641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:38.864601 4951 patch_prober.go:29] interesting pod/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 07:39:38.865276 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:38.865242 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" podUID="e1304cf8-fc22-48db-acce-17514c25cf9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 07:39:39.851078 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:39.851038 4951 patch_prober.go:29] interesting pod/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 07:39:39.851405 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:39.851373 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" podUID="e1304cf8-fc22-48db-acce-17514c25cf9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 07:39:40.986974 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:40.986945 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:39:40.988457 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:40.988430 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 07:39:48.097227 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:48.097190 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg"] Oct 30 07:39:48.097689 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:48.097416 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" podUID="d493c768-747e-40e3-8e65-1c7fcb582c23" containerName="istio-proxy" containerID="cri-o://a9e7cb0222c9aaa92cd97fd7012cb17b60cb8333fceca0d6d6919b45fec8d32e" gracePeriod=30 Oct 30 07:39:51.892267 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:51.892227 4951 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:39:51.892684 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:51.892296 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" podUID="d493c768-747e-40e3-8e65-1c7fcb582c23" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:39:53.025627 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:39:53.025582 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:39:53.298225 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:53.298185 4951 generic.go:356] "Generic (PLEG): container finished" podID="d493c768-747e-40e3-8e65-1c7fcb582c23" containerID="a9e7cb0222c9aaa92cd97fd7012cb17b60cb8333fceca0d6d6919b45fec8d32e" exitCode=0 Oct 30 07:39:53.298688 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:53.298242 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" event={"ID":"d493c768-747e-40e3-8e65-1c7fcb582c23","Type":"ContainerDied","Data":"a9e7cb0222c9aaa92cd97fd7012cb17b60cb8333fceca0d6d6919b45fec8d32e"} Oct 30 07:39:53.704333 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:53.704308 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:39:53.841438 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:53.841402 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-workload-socket\") pod \"d493c768-747e-40e3-8e65-1c7fcb582c23\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " Oct 30 07:39:53.841770 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:53.841739 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-credential-socket\") pod \"d493c768-747e-40e3-8e65-1c7fcb582c23\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " Oct 30 07:39:53.841932 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:53.841917 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-istio-envoy\") pod \"d493c768-747e-40e3-8e65-1c7fcb582c23\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " Oct 30 07:39:53.842115 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:53.842099 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d493c768-747e-40e3-8e65-1c7fcb582c23-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d493c768-747e-40e3-8e65-1c7fcb582c23-istiod-ca-cert\") pod \"d493c768-747e-40e3-8e65-1c7fcb582c23\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " Oct 30 07:39:53.842250 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:53.842235 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d493c768-747e-40e3-8e65-1c7fcb582c23-istio-token\" (UniqueName: \"kubernetes.io/projected/d493c768-747e-40e3-8e65-1c7fcb582c23-istio-token\") pod \"d493c768-747e-40e3-8e65-1c7fcb582c23\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " Oct 30 07:39:53.842438 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:53.842422 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-istio-data\") pod \"d493c768-747e-40e3-8e65-1c7fcb582c23\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " Oct 30 07:39:53.842647 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:53.842628 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d493c768-747e-40e3-8e65-1c7fcb582c23-kube-api-access-rzh97\" (UniqueName: \"kubernetes.io/projected/d493c768-747e-40e3-8e65-1c7fcb582c23-kube-api-access-rzh97\") pod \"d493c768-747e-40e3-8e65-1c7fcb582c23\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " Oct 30 07:39:53.842911 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:53.842893 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-workload-certs\") pod \"d493c768-747e-40e3-8e65-1c7fcb582c23\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " Oct 30 07:39:53.843038 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:53.843020 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d493c768-747e-40e3-8e65-1c7fcb582c23-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d493c768-747e-40e3-8e65-1c7fcb582c23-istio-podinfo\") pod \"d493c768-747e-40e3-8e65-1c7fcb582c23\" (UID: \"d493c768-747e-40e3-8e65-1c7fcb582c23\") " Oct 30 07:39:53.844645 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:53.844619 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d493c768-747e-40e3-8e65-1c7fcb582c23-istiod-ca-cert" pod "d493c768-747e-40e3-8e65-1c7fcb582c23" (UID: "d493c768-747e-40e3-8e65-1c7fcb582c23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:39:53.849406 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:53.849380 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-workload-socket" pod "d493c768-747e-40e3-8e65-1c7fcb582c23" (UID: "d493c768-747e-40e3-8e65-1c7fcb582c23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:39:53.849815 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:53.849795 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-credential-socket" pod "d493c768-747e-40e3-8e65-1c7fcb582c23" (UID: "d493c768-747e-40e3-8e65-1c7fcb582c23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:39:53.850686 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:53.850398 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-istio-data" pod "d493c768-747e-40e3-8e65-1c7fcb582c23" (UID: "d493c768-747e-40e3-8e65-1c7fcb582c23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:39:53.891845 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:53.891718 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-workload-certs" pod "d493c768-747e-40e3-8e65-1c7fcb582c23" (UID: "d493c768-747e-40e3-8e65-1c7fcb582c23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:39:53.899990 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:53.899780 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d493c768-747e-40e3-8e65-1c7fcb582c23-istio-token" pod "d493c768-747e-40e3-8e65-1c7fcb582c23" (UID: "d493c768-747e-40e3-8e65-1c7fcb582c23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:39:53.905701 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:53.900317 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d493c768-747e-40e3-8e65-1c7fcb582c23-istio-podinfo" pod "d493c768-747e-40e3-8e65-1c7fcb582c23" (UID: "d493c768-747e-40e3-8e65-1c7fcb582c23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:39:53.913835 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:53.913799 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-istio-envoy" pod "d493c768-747e-40e3-8e65-1c7fcb582c23" (UID: "d493c768-747e-40e3-8e65-1c7fcb582c23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:39:53.914951 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:53.914670 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d493c768-747e-40e3-8e65-1c7fcb582c23-kube-api-access-rzh97" pod "d493c768-747e-40e3-8e65-1c7fcb582c23" (UID: "d493c768-747e-40e3-8e65-1c7fcb582c23"). InnerVolumeSpecName "kube-api-access-rzh97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:39:53.945724 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:53.945696 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:39:53.946052 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:53.946032 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:39:53.946172 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:53.946161 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d493c768-747e-40e3-8e65-1c7fcb582c23-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:39:53.946262 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:53.946254 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d493c768-747e-40e3-8e65-1c7fcb582c23-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:39:53.946320 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:53.946314 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:39:53.946380 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:53.946372 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzh97\" (UniqueName: \"kubernetes.io/projected/d493c768-747e-40e3-8e65-1c7fcb582c23-kube-api-access-rzh97\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:39:53.946434 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:53.946425 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:39:53.946484 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:53.946478 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d493c768-747e-40e3-8e65-1c7fcb582c23-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:39:53.946534 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:53.946527 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d493c768-747e-40e3-8e65-1c7fcb582c23-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:39:54.308746 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:54.308704 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" Oct 30 07:39:54.308746 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:54.308723 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg" event={"ID":"d493c768-747e-40e3-8e65-1c7fcb582c23","Type":"ContainerDied","Data":"58de8276b0c5b9d43240017721745c87d53cc261c1034d874316ae5a60a03346"} Oct 30 07:39:54.309355 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:54.308803 4951 scope.go:122] "RemoveContainer" containerID="a9e7cb0222c9aaa92cd97fd7012cb17b60cb8333fceca0d6d6919b45fec8d32e" Oct 30 07:39:54.371187 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:54.371150 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg"] Oct 30 07:39:54.381800 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:54.381768 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-rwhzg"] Oct 30 07:39:55.774081 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:39:55.774035 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d493c768-747e-40e3-8e65-1c7fcb582c23" path="/var/lib/kubelet/pods/d493c768-747e-40e3-8e65-1c7fcb582c23/volumes" Oct 30 07:40:14.311119 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:14.311084 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk"] Oct 30 07:40:14.312098 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:14.312070 4951 cpu_manager.go:414] "containerMap: removing container" podUID="d493c768-747e-40e3-8e65-1c7fcb582c23" containerName="istio-proxy" Oct 30 07:40:14.312098 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:14.312095 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d493c768-747e-40e3-8e65-1c7fcb582c23" containerName="istio-proxy" Oct 30 07:40:14.312322 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:14.312308 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="d493c768-747e-40e3-8e65-1c7fcb582c23" containerName="istio-proxy" Oct 30 07:40:14.315950 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:14.315931 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk" Oct 30 07:40:14.336813 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:14.336786 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-bb2e8f-serving-cert\"" Oct 30 07:40:14.343360 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:14.343332 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk"] Oct 30 07:40:14.388111 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:14.388059 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5hbb\" (UniqueName: \"kubernetes.io/projected/92a44e19-26dc-46b2-8a2b-5d5a33941087-kube-api-access-c5hbb\") pod \"maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk\" (UID: \"92a44e19-26dc-46b2-8a2b-5d5a33941087\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk" Oct 30 07:40:14.388325 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:14.388249 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92a44e19-26dc-46b2-8a2b-5d5a33941087-maas-api-tls\") pod \"maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk\" (UID: \"92a44e19-26dc-46b2-8a2b-5d5a33941087\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk" Oct 30 07:40:14.388413 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:14.388368 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92a44e19-26dc-46b2-8a2b-5d5a33941087-metrics-certs\") pod \"maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk\" (UID: \"92a44e19-26dc-46b2-8a2b-5d5a33941087\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk" Oct 30 07:40:14.388532 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:14.388506 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92a44e19-26dc-46b2-8a2b-5d5a33941087-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk\" (UID: \"92a44e19-26dc-46b2-8a2b-5d5a33941087\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk" Oct 30 07:40:14.489461 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:14.489413 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92a44e19-26dc-46b2-8a2b-5d5a33941087-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk\" (UID: \"92a44e19-26dc-46b2-8a2b-5d5a33941087\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk" Oct 30 07:40:14.489710 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:14.489498 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5hbb\" (UniqueName: \"kubernetes.io/projected/92a44e19-26dc-46b2-8a2b-5d5a33941087-kube-api-access-c5hbb\") pod \"maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk\" (UID: \"92a44e19-26dc-46b2-8a2b-5d5a33941087\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk" Oct 30 07:40:14.489710 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:14.489646 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92a44e19-26dc-46b2-8a2b-5d5a33941087-maas-api-tls\") pod \"maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk\" (UID: \"92a44e19-26dc-46b2-8a2b-5d5a33941087\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk" Oct 30 07:40:14.489806 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:14.489763 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92a44e19-26dc-46b2-8a2b-5d5a33941087-metrics-certs\") pod \"maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk\" (UID: \"92a44e19-26dc-46b2-8a2b-5d5a33941087\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk" Oct 30 07:40:14.490137 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:14.490111 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92a44e19-26dc-46b2-8a2b-5d5a33941087-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk\" (UID: \"92a44e19-26dc-46b2-8a2b-5d5a33941087\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk" Oct 30 07:40:14.492190 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:14.492163 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92a44e19-26dc-46b2-8a2b-5d5a33941087-metrics-certs\") pod \"maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk\" (UID: \"92a44e19-26dc-46b2-8a2b-5d5a33941087\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk" Oct 30 07:40:14.492308 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:14.492192 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92a44e19-26dc-46b2-8a2b-5d5a33941087-maas-api-tls\") pod \"maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk\" (UID: \"92a44e19-26dc-46b2-8a2b-5d5a33941087\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk" Oct 30 07:40:14.540596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:14.537786 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5hbb\" (UniqueName: \"kubernetes.io/projected/92a44e19-26dc-46b2-8a2b-5d5a33941087-kube-api-access-c5hbb\") pod \"maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk\" (UID: \"92a44e19-26dc-46b2-8a2b-5d5a33941087\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk" Oct 30 07:40:14.632429 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:14.632341 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk" Oct 30 07:40:14.943829 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:40:14.943787 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a44e19_26dc_46b2_8a2b_5d5a33941087.slice/crio-28cbe79cf483f13f4b332a48f31405386dfd0394222f1026b52cbb62038a891c WatchSource:0}: Error finding container 28cbe79cf483f13f4b332a48f31405386dfd0394222f1026b52cbb62038a891c: Status 404 returned error can't find the container with id 28cbe79cf483f13f4b332a48f31405386dfd0394222f1026b52cbb62038a891c Oct 30 07:40:14.945195 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:14.945166 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk"] Oct 30 07:40:15.523954 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:15.523918 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk" event={"ID":"92a44e19-26dc-46b2-8a2b-5d5a33941087","Type":"ContainerStarted","Data":"754c26b3e50f88bb715e21fa391e5454996543d40f83f00364b6834c26ba501b"} Oct 30 07:40:15.524524 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:15.524500 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk" event={"ID":"92a44e19-26dc-46b2-8a2b-5d5a33941087","Type":"ContainerStarted","Data":"28cbe79cf483f13f4b332a48f31405386dfd0394222f1026b52cbb62038a891c"} Oct 30 07:40:15.525844 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:15.525822 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk" Oct 30 07:40:15.572231 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:15.572131 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk" podStartSLOduration=1.227137382 podStartE2EDuration="1.57211307s" totalImagesPullingTime="344.975688ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:40:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:40:15.571784271 +0000 UTC m=+2621.914832236" watchObservedRunningTime="2026-10-30 07:40:15.57211307 +0000 UTC m=+2621.915161034" Oct 30 07:40:22.573612 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:22.573538 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk" Oct 30 07:40:23.106249 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:40:23.106202 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:40:27.846046 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:27.846007 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4"] Oct 30 07:40:27.852419 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:27.852389 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 07:40:27.879264 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:27.879160 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-705337-openshift-default-dockercfg-z72xk\"" Oct 30 07:40:27.889894 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:27.889856 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4"] Oct 30 07:40:27.972785 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:27.972752 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-workload-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 07:40:27.972961 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:27.972795 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qncbv\" (UniqueName: \"kubernetes.io/projected/ef44a41b-a51b-49e4-aa7d-ae5ece873299-kube-api-access-qncbv\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 07:40:27.972961 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:27.972929 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istio-envoy\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 07:40:27.973056 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:27.972963 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istiod-ca-cert\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 07:40:27.973056 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:27.973036 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-workload-certs\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 07:40:27.973128 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:27.973110 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istio-data\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 07:40:27.973163 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:27.973149 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istio-token\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 07:40:27.973240 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:27.973215 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istio-podinfo\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 07:40:27.973289 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:27.973277 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-credential-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 07:40:28.075137 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:28.075106 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-workload-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 07:40:28.075137 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:28.075140 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qncbv\" (UniqueName: \"kubernetes.io/projected/ef44a41b-a51b-49e4-aa7d-ae5ece873299-kube-api-access-qncbv\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 07:40:28.075355 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:28.075199 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istio-envoy\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 07:40:28.075355 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:28.075218 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istiod-ca-cert\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 07:40:28.075355 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:28.075285 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-workload-certs\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 07:40:28.075355 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:28.075336 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istio-data\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 07:40:28.075560 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:28.075386 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istio-token\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 07:40:28.075560 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:28.075436 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istio-podinfo\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 07:40:28.075560 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:28.075493 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-credential-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 07:40:28.075786 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:28.075588 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-workload-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 07:40:28.075786 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:28.075686 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-workload-certs\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 07:40:28.075907 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:28.075831 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istio-data\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 07:40:28.076016 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:28.075995 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-credential-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 07:40:28.076056 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:28.076018 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istiod-ca-cert\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 07:40:28.077613 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:28.077589 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istio-envoy\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 07:40:28.077931 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:28.077914 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istio-podinfo\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 07:40:28.100347 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:28.100288 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istio-token\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 07:40:28.100491 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:28.100476 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qncbv\" (UniqueName: \"kubernetes.io/projected/ef44a41b-a51b-49e4-aa7d-ae5ece873299-kube-api-access-qncbv\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 07:40:28.170379 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:28.170352 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 07:40:28.494346 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:40:28.494309 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef44a41b_a51b_49e4_aa7d_ae5ece873299.slice/crio-19ff653ace0b35c2476fd250ca340a354c0acb0fe6c80d0cb229e23cf57f694b WatchSource:0}: Error finding container 19ff653ace0b35c2476fd250ca340a354c0acb0fe6c80d0cb229e23cf57f694b: Status 404 returned error can't find the container with id 19ff653ace0b35c2476fd250ca340a354c0acb0fe6c80d0cb229e23cf57f694b Oct 30 07:40:28.495680 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:28.495653 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4"] Oct 30 07:40:28.497019 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:28.496984 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:40:28.497079 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:28.497062 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:40:28.497116 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:28.497091 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:40:28.735679 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:28.735636 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" event={"ID":"ef44a41b-a51b-49e4-aa7d-ae5ece873299","Type":"ContainerStarted","Data":"8a9b08338a6ad1e49c6765e87338eaa4ce72fb71c5a6d3d8477aa100251f515f"} Oct 30 07:40:28.735879 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:28.735707 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" event={"ID":"ef44a41b-a51b-49e4-aa7d-ae5ece873299","Type":"ContainerStarted","Data":"19ff653ace0b35c2476fd250ca340a354c0acb0fe6c80d0cb229e23cf57f694b"} Oct 30 07:40:28.787262 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:28.787198 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" podStartSLOduration=1.787179334 podStartE2EDuration="1.787179334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:40:28.784625862 +0000 UTC m=+2635.127673825" watchObservedRunningTime="2026-10-30 07:40:28.787179334 +0000 UTC m=+2635.130227299" Oct 30 07:40:29.170842 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:29.170750 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 07:40:29.172737 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:29.172707 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 30 07:40:29.172871 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:29.172767 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" podUID="ef44a41b-a51b-49e4-aa7d-ae5ece873299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 30 07:40:30.175596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:30.175008 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 30 07:40:30.175596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:30.175086 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" podUID="ef44a41b-a51b-49e4-aa7d-ae5ece873299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 30 07:40:31.170969 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:31.170935 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 30 07:40:31.171154 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:31.170988 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" podUID="ef44a41b-a51b-49e4-aa7d-ae5ece873299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 30 07:40:32.171272 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:32.171230 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 30 07:40:32.171736 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:32.171312 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" podUID="ef44a41b-a51b-49e4-aa7d-ae5ece873299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 30 07:40:33.171411 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:33.171368 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 30 07:40:33.171971 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:33.171433 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" podUID="ef44a41b-a51b-49e4-aa7d-ae5ece873299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 30 07:40:34.171341 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:34.171308 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 30 07:40:34.171566 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:34.171377 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" podUID="ef44a41b-a51b-49e4-aa7d-ae5ece873299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 30 07:40:35.171798 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:35.171761 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Oct 30 07:40:35.172209 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:35.171823 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" podUID="ef44a41b-a51b-49e4-aa7d-ae5ece873299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Oct 30 07:40:36.176228 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:36.176195 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 07:40:36.176677 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:36.176664 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 07:40:36.177639 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:36.177614 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 07:40:42.897113 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:42.897056 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29"] Oct 30 07:40:42.909686 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:42.909655 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:40:42.919174 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:42.919128 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29"] Oct 30 07:40:42.941021 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:42.940984 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh"] Oct 30 07:40:42.984917 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:42.984882 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh"] Oct 30 07:40:42.985115 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:42.985006 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:40:43.070553 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.070513 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istio-envoy\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:40:43.070805 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.070785 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flvrx\" (UniqueName: \"kubernetes.io/projected/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-kube-api-access-flvrx\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:40:43.071107 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.071078 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istio-token\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:40:43.071171 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.071153 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-credential-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:40:43.071214 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.071201 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istiod-ca-cert\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:40:43.071250 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.071232 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istio-podinfo\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:40:43.071302 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.071280 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istio-data\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:40:43.071395 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.071381 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-workload-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:40:43.071478 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.071464 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-workload-certs\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:40:43.173379 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.173291 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32fb4483-9210-4a96-8296-9354fa752449-istio-podinfo\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:40:43.173596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.173415 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32fb4483-9210-4a96-8296-9354fa752449-istiod-ca-cert\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:40:43.173596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.173473 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istio-token\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:40:43.173596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.173513 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-workload-certs\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:40:43.173804 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.173695 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk8k8\" (UniqueName: \"kubernetes.io/projected/32fb4483-9210-4a96-8296-9354fa752449-kube-api-access-vk8k8\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:40:43.173804 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.173744 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-credential-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:40:43.173897 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.173819 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istiod-ca-cert\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:40:43.173897 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.173859 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istio-podinfo\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:40:43.174012 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.173902 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istio-data\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:40:43.174012 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.173977 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32fb4483-9210-4a96-8296-9354fa752449-istio-token\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:40:43.174130 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.174018 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-workload-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:40:43.174130 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.174068 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-workload-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:40:43.174130 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.174092 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-istio-data\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:40:43.174304 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.174139 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-workload-certs\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:40:43.174304 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.174166 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-credential-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:40:43.174304 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.174174 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-credential-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:40:43.174467 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.174422 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istio-envoy\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:40:43.174529 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.174508 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istio-data\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:40:43.174615 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.174545 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flvrx\" (UniqueName: \"kubernetes.io/projected/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-kube-api-access-flvrx\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:40:43.174748 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.174721 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-istio-envoy\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:40:43.174853 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.174762 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-workload-certs\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:40:43.174853 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.174812 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istiod-ca-cert\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:40:43.175229 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.175199 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-workload-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:40:43.176924 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.176903 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istio-envoy\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:40:43.177388 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.177363 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istio-podinfo\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:40:43.204590 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.204051 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istio-token\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:40:43.204590 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.204378 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flvrx\" (UniqueName: \"kubernetes.io/projected/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-kube-api-access-flvrx\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:40:43.276247 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.276210 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32fb4483-9210-4a96-8296-9354fa752449-istio-token\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:40:43.276435 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.276370 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-workload-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:40:43.276435 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.276405 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-istio-data\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:40:43.276531 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.276451 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-credential-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:40:43.276531 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.276525 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-istio-envoy\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:40:43.276645 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.276598 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32fb4483-9210-4a96-8296-9354fa752449-istio-podinfo\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:40:43.276724 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.276667 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32fb4483-9210-4a96-8296-9354fa752449-istiod-ca-cert\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:40:43.276783 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.276722 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-workload-certs\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:40:43.276783 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.276764 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk8k8\" (UniqueName: \"kubernetes.io/projected/32fb4483-9210-4a96-8296-9354fa752449-kube-api-access-vk8k8\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:40:43.276896 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.276791 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-workload-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:40:43.276896 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.276833 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-istio-data\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:40:43.277000 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.276922 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-credential-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:40:43.277096 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.277074 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-workload-certs\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:40:43.277451 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.277428 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32fb4483-9210-4a96-8296-9354fa752449-istiod-ca-cert\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:40:43.279063 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.279039 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-istio-envoy\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:40:43.279199 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.279181 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32fb4483-9210-4a96-8296-9354fa752449-istio-podinfo\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:40:43.290364 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.290334 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:40:43.316254 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.316221 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32fb4483-9210-4a96-8296-9354fa752449-istio-token\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:40:43.354790 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.354644 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk8k8\" (UniqueName: \"kubernetes.io/projected/32fb4483-9210-4a96-8296-9354fa752449-kube-api-access-vk8k8\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:40:43.611412 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.611382 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:40:43.667185 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.667155 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29"] Oct 30 07:40:43.669465 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:40:43.669427 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a75b3c1_7b1a_47aa_ad25_3d16a3e5babc.slice/crio-e47408096503beb3ff8fee081e8334db5b3524d852cac9ac9cff10fbc3301ccb WatchSource:0}: Error finding container e47408096503beb3ff8fee081e8334db5b3524d852cac9ac9cff10fbc3301ccb: Status 404 returned error can't find the container with id e47408096503beb3ff8fee081e8334db5b3524d852cac9ac9cff10fbc3301ccb Oct 30 07:40:43.674489 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.674450 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:40:43.674580 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.674541 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:40:43.674629 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.674600 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:40:43.979773 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:40:43.979716 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32fb4483_9210_4a96_8296_9354fa752449.slice/crio-364b63718576bd6dc3c68fc5281f7ea21c2b6c501fcffb3f250b59259f0da11c WatchSource:0}: Error finding container 364b63718576bd6dc3c68fc5281f7ea21c2b6c501fcffb3f250b59259f0da11c: Status 404 returned error can't find the container with id 364b63718576bd6dc3c68fc5281f7ea21c2b6c501fcffb3f250b59259f0da11c Oct 30 07:40:43.981192 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.981159 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh"] Oct 30 07:40:43.984157 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.984108 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:40:43.984280 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.984206 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:40:43.984280 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:43.984268 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:40:44.187991 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:44.187952 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" event={"ID":"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc","Type":"ContainerStarted","Data":"95fc409e40a29891a0a7ba98ea868b4c640d4b878f637ed1fca814997fc5e690"} Oct 30 07:40:44.188164 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:44.188010 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" event={"ID":"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc","Type":"ContainerStarted","Data":"e47408096503beb3ff8fee081e8334db5b3524d852cac9ac9cff10fbc3301ccb"} Oct 30 07:40:44.190814 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:44.190787 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" event={"ID":"32fb4483-9210-4a96-8296-9354fa752449","Type":"ContainerStarted","Data":"3e57c429da62714aab3b0cd76beb27e04afadded253c74b6313c6855b527dfba"} Oct 30 07:40:44.191033 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:44.191018 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" event={"ID":"32fb4483-9210-4a96-8296-9354fa752449","Type":"ContainerStarted","Data":"364b63718576bd6dc3c68fc5281f7ea21c2b6c501fcffb3f250b59259f0da11c"} Oct 30 07:40:44.246647 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:44.245918 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" podStartSLOduration=2.245898945 podStartE2EDuration="2.245898945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:40:44.245067634 +0000 UTC m=+2650.588115616" watchObservedRunningTime="2026-10-30 07:40:44.245898945 +0000 UTC m=+2650.588946914" Oct 30 07:40:44.292063 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:44.292035 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:40:44.293851 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:44.293817 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Oct 30 07:40:44.293983 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:44.293873 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" podUID="3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Oct 30 07:40:44.307098 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:44.307028 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" podStartSLOduration=2.306362531 podStartE2EDuration="2.306362531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:40:44.298716439 +0000 UTC m=+2650.641764405" watchObservedRunningTime="2026-10-30 07:40:44.306362531 +0000 UTC m=+2650.649410497" Oct 30 07:40:44.612236 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:44.612201 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:40:45.303387 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:45.303348 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Oct 30 07:40:45.304099 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:45.304065 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" podUID="3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Oct 30 07:40:45.614837 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:45.614710 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 30 07:40:45.614837 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:45.614763 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" podUID="32fb4483-9210-4a96-8296-9354fa752449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 30 07:40:46.297711 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:46.293439 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Oct 30 07:40:46.297711 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:46.293494 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" podUID="3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Oct 30 07:40:46.616668 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:46.616015 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 30 07:40:46.616668 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:46.616073 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" podUID="32fb4483-9210-4a96-8296-9354fa752449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 30 07:40:47.292800 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:47.292331 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Oct 30 07:40:47.292800 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:47.292399 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" podUID="3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Oct 30 07:40:47.612657 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:47.612616 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 30 07:40:47.612907 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:47.612880 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" podUID="32fb4483-9210-4a96-8296-9354fa752449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 30 07:40:48.330489 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:48.330065 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Oct 30 07:40:48.330489 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:48.330121 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" podUID="3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Oct 30 07:40:48.612023 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:48.611935 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 30 07:40:48.612023 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:48.611993 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" podUID="32fb4483-9210-4a96-8296-9354fa752449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 30 07:40:49.291532 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:49.291485 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Oct 30 07:40:49.291771 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:49.291562 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" podUID="3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Oct 30 07:40:49.612467 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:49.612379 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 30 07:40:49.612956 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:49.612460 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" podUID="32fb4483-9210-4a96-8296-9354fa752449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 30 07:40:50.294145 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:50.294106 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Oct 30 07:40:50.294474 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:50.294441 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" podUID="3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Oct 30 07:40:50.612477 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:50.611923 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 30 07:40:50.612477 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:50.612004 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" podUID="32fb4483-9210-4a96-8296-9354fa752449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 30 07:40:51.295820 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:51.295783 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:40:51.296329 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:51.296300 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:40:51.297007 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:51.296988 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:40:51.617953 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:51.617878 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:40:51.618438 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:51.618421 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:40:51.619168 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:51.619150 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:40:53.025859 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:40:53.025815 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:40:54.427287 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:54.427250 4951 generic.go:356] "Generic (PLEG): container finished" podID="b08b669b-c0a7-462c-9b63-aabbb0254dc1" containerID="dfe77b3f6c8dbc03e45aaa35620dc0846e8baba105df5275450257231527b35d" exitCode=0 Oct 30 07:40:54.427984 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:54.427328 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" event={"ID":"b08b669b-c0a7-462c-9b63-aabbb0254dc1","Type":"ContainerDied","Data":"dfe77b3f6c8dbc03e45aaa35620dc0846e8baba105df5275450257231527b35d"} Oct 30 07:40:54.427984 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:54.427876 4951 scope.go:122] "RemoveContainer" containerID="dfe77b3f6c8dbc03e45aaa35620dc0846e8baba105df5275450257231527b35d" Oct 30 07:40:54.429833 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:54.429800 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:40:54.429948 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:54.429868 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:40:54.429948 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:54.429906 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:40:55.457940 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:55.457870 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" event={"ID":"b08b669b-c0a7-462c-9b63-aabbb0254dc1","Type":"ContainerStarted","Data":"4d0bd6310830aebfaa316fa880e9568454ffeb3bc27c3f2b2a41f603cc76ad57"} Oct 30 07:40:55.492012 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:55.491979 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:40:55.493655 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:55.493624 4951 patch_prober.go:29] interesting pod/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 07:40:55.493819 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:55.493683 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" podUID="b08b669b-c0a7-462c-9b63-aabbb0254dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 07:40:56.491777 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:56.491741 4951 patch_prober.go:29] interesting pod/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 07:40:56.492180 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:56.491793 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" podUID="b08b669b-c0a7-462c-9b63-aabbb0254dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 07:40:57.491653 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:57.491610 4951 patch_prober.go:29] interesting pod/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 07:40:57.491874 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:57.491683 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" podUID="b08b669b-c0a7-462c-9b63-aabbb0254dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 07:40:58.495594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:58.492144 4951 patch_prober.go:29] interesting pod/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 07:40:58.495594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:58.492224 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" podUID="b08b669b-c0a7-462c-9b63-aabbb0254dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 07:40:59.492001 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:59.491963 4951 patch_prober.go:29] interesting pod/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 07:40:59.492277 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:59.492023 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" podUID="b08b669b-c0a7-462c-9b63-aabbb0254dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 07:40:59.968100 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:40:59.968068 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:41:00.491738 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:41:00.491703 4951 patch_prober.go:29] interesting pod/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 07:41:00.491958 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:41:00.491764 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" podUID="b08b669b-c0a7-462c-9b63-aabbb0254dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 07:41:01.498017 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:41:01.497981 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:41:01.499328 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:41:01.499305 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 07:41:23.027544 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:41:23.027500 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:41:34.502959 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:41:34.502926 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/3.log" Oct 30 07:41:34.508832 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:41:34.508807 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 07:41:34.515751 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:41:34.515727 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/3.log" Oct 30 07:41:34.521459 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:41:34.521441 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 07:41:53.024349 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:41:53.024309 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:42:23.024777 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:42:23.024693 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:42:53.031784 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:42:53.031744 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:43:02.161005 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.160963 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg"] Oct 30 07:43:02.166470 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.166440 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" Oct 30 07:43:02.183392 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.183363 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-j9rfz\"" Oct 30 07:43:02.202898 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.202846 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg"] Oct 30 07:43:02.291112 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.291067 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9e850f-e9fb-4aa6-ba8d-4113648338e3-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-b2nrg\" (UID: \"6c9e850f-e9fb-4aa6-ba8d-4113648338e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" Oct 30 07:43:02.291329 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.291196 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9e850f-e9fb-4aa6-ba8d-4113648338e3-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-b2nrg\" (UID: \"6c9e850f-e9fb-4aa6-ba8d-4113648338e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" Oct 30 07:43:02.291329 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.291283 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c9e850f-e9fb-4aa6-ba8d-4113648338e3-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-b2nrg\" (UID: \"6c9e850f-e9fb-4aa6-ba8d-4113648338e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" Oct 30 07:43:02.291466 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.291448 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c9e850f-e9fb-4aa6-ba8d-4113648338e3-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-b2nrg\" (UID: \"6c9e850f-e9fb-4aa6-ba8d-4113648338e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" Oct 30 07:43:02.291527 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.291512 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c9e850f-e9fb-4aa6-ba8d-4113648338e3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-b2nrg\" (UID: \"6c9e850f-e9fb-4aa6-ba8d-4113648338e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" Oct 30 07:43:02.291627 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.291544 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5c7k\" (UniqueName: \"kubernetes.io/projected/6c9e850f-e9fb-4aa6-ba8d-4113648338e3-kube-api-access-p5c7k\") pod \"maas-default-gateway-openshift-default-6dcf757899-b2nrg\" (UID: \"6c9e850f-e9fb-4aa6-ba8d-4113648338e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" Oct 30 07:43:02.291681 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.291654 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c9e850f-e9fb-4aa6-ba8d-4113648338e3-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-b2nrg\" (UID: \"6c9e850f-e9fb-4aa6-ba8d-4113648338e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" Oct 30 07:43:02.291758 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.291744 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c9e850f-e9fb-4aa6-ba8d-4113648338e3-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-b2nrg\" (UID: \"6c9e850f-e9fb-4aa6-ba8d-4113648338e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" Oct 30 07:43:02.291819 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.291806 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c9e850f-e9fb-4aa6-ba8d-4113648338e3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-b2nrg\" (UID: \"6c9e850f-e9fb-4aa6-ba8d-4113648338e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" Oct 30 07:43:02.396955 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.394754 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c9e850f-e9fb-4aa6-ba8d-4113648338e3-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-b2nrg\" (UID: \"6c9e850f-e9fb-4aa6-ba8d-4113648338e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" Oct 30 07:43:02.396955 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.394941 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c9e850f-e9fb-4aa6-ba8d-4113648338e3-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-b2nrg\" (UID: \"6c9e850f-e9fb-4aa6-ba8d-4113648338e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" Oct 30 07:43:02.396955 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.394990 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c9e850f-e9fb-4aa6-ba8d-4113648338e3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-b2nrg\" (UID: \"6c9e850f-e9fb-4aa6-ba8d-4113648338e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" Oct 30 07:43:02.396955 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.395050 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9e850f-e9fb-4aa6-ba8d-4113648338e3-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-b2nrg\" (UID: \"6c9e850f-e9fb-4aa6-ba8d-4113648338e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" Oct 30 07:43:02.396955 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.395127 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9e850f-e9fb-4aa6-ba8d-4113648338e3-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-b2nrg\" (UID: \"6c9e850f-e9fb-4aa6-ba8d-4113648338e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" Oct 30 07:43:02.396955 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.395199 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c9e850f-e9fb-4aa6-ba8d-4113648338e3-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-b2nrg\" (UID: \"6c9e850f-e9fb-4aa6-ba8d-4113648338e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" Oct 30 07:43:02.396955 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.395312 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c9e850f-e9fb-4aa6-ba8d-4113648338e3-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-b2nrg\" (UID: \"6c9e850f-e9fb-4aa6-ba8d-4113648338e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" Oct 30 07:43:02.396955 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.395370 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c9e850f-e9fb-4aa6-ba8d-4113648338e3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-b2nrg\" (UID: \"6c9e850f-e9fb-4aa6-ba8d-4113648338e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" Oct 30 07:43:02.396955 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.395398 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5c7k\" (UniqueName: \"kubernetes.io/projected/6c9e850f-e9fb-4aa6-ba8d-4113648338e3-kube-api-access-p5c7k\") pod \"maas-default-gateway-openshift-default-6dcf757899-b2nrg\" (UID: \"6c9e850f-e9fb-4aa6-ba8d-4113648338e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" Oct 30 07:43:02.396955 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.396258 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c9e850f-e9fb-4aa6-ba8d-4113648338e3-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-b2nrg\" (UID: \"6c9e850f-e9fb-4aa6-ba8d-4113648338e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" Oct 30 07:43:02.396955 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.396519 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c9e850f-e9fb-4aa6-ba8d-4113648338e3-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-b2nrg\" (UID: \"6c9e850f-e9fb-4aa6-ba8d-4113648338e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" Oct 30 07:43:02.406592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.404556 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9e850f-e9fb-4aa6-ba8d-4113648338e3-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-b2nrg\" (UID: \"6c9e850f-e9fb-4aa6-ba8d-4113648338e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" Oct 30 07:43:02.406592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.404839 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9e850f-e9fb-4aa6-ba8d-4113648338e3-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-b2nrg\" (UID: \"6c9e850f-e9fb-4aa6-ba8d-4113648338e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" Oct 30 07:43:02.406592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.406332 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c9e850f-e9fb-4aa6-ba8d-4113648338e3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-b2nrg\" (UID: \"6c9e850f-e9fb-4aa6-ba8d-4113648338e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" Oct 30 07:43:02.417914 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.417850 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c9e850f-e9fb-4aa6-ba8d-4113648338e3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-b2nrg\" (UID: \"6c9e850f-e9fb-4aa6-ba8d-4113648338e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" Oct 30 07:43:02.430702 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.421362 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5c7k\" (UniqueName: \"kubernetes.io/projected/6c9e850f-e9fb-4aa6-ba8d-4113648338e3-kube-api-access-p5c7k\") pod \"maas-default-gateway-openshift-default-6dcf757899-b2nrg\" (UID: \"6c9e850f-e9fb-4aa6-ba8d-4113648338e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" Oct 30 07:43:02.451957 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.451916 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c9e850f-e9fb-4aa6-ba8d-4113648338e3-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-b2nrg\" (UID: \"6c9e850f-e9fb-4aa6-ba8d-4113648338e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" Oct 30 07:43:02.465670 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.452833 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c9e850f-e9fb-4aa6-ba8d-4113648338e3-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-b2nrg\" (UID: \"6c9e850f-e9fb-4aa6-ba8d-4113648338e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" Oct 30 07:43:02.487237 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.486914 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" Oct 30 07:43:02.901091 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:43:02.901057 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c9e850f_e9fb_4aa6_ba8d_4113648338e3.slice/crio-07e8c2bc88db1a93e516e8bceb65311b99753a0a7c01872bd4aa3e911bcaa192 WatchSource:0}: Error finding container 07e8c2bc88db1a93e516e8bceb65311b99753a0a7c01872bd4aa3e911bcaa192: Status 404 returned error can't find the container with id 07e8c2bc88db1a93e516e8bceb65311b99753a0a7c01872bd4aa3e911bcaa192 Oct 30 07:43:02.902657 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.902630 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg"] Oct 30 07:43:02.904117 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.904085 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:43:02.904182 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.904151 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:43:02.904223 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:02.904181 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:43:03.245243 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:03.245146 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" event={"ID":"6c9e850f-e9fb-4aa6-ba8d-4113648338e3","Type":"ContainerStarted","Data":"01a736edb610d7319c9f685bf2d8a43242522d6805f025a558e784406bd480d3"} Oct 30 07:43:03.245243 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:03.245200 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" event={"ID":"6c9e850f-e9fb-4aa6-ba8d-4113648338e3","Type":"ContainerStarted","Data":"07e8c2bc88db1a93e516e8bceb65311b99753a0a7c01872bd4aa3e911bcaa192"} Oct 30 07:43:03.359595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:03.359185 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" podStartSLOduration=1.359162784 podStartE2EDuration="1.359162784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:43:03.300283833 +0000 UTC m=+2789.643331839" watchObservedRunningTime="2026-10-30 07:43:03.359162784 +0000 UTC m=+2789.702210751" Oct 30 07:43:03.491241 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:03.490740 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" Oct 30 07:43:03.494776 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:03.494738 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-b2nrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 07:43:03.494922 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:03.494813 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" podUID="6c9e850f-e9fb-4aa6-ba8d-4113648338e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 07:43:04.489064 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:04.489014 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-b2nrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 07:43:04.489786 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:04.489746 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" podUID="6c9e850f-e9fb-4aa6-ba8d-4113648338e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 07:43:05.488700 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:05.488640 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-b2nrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 07:43:05.488925 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:05.488721 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" podUID="6c9e850f-e9fb-4aa6-ba8d-4113648338e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 07:43:06.491600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:06.488951 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-b2nrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 07:43:06.491600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:06.489013 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" podUID="6c9e850f-e9fb-4aa6-ba8d-4113648338e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 07:43:07.487863 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:07.487817 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-b2nrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 07:43:07.487863 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:07.487878 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" podUID="6c9e850f-e9fb-4aa6-ba8d-4113648338e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 07:43:08.493606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:08.490598 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-b2nrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 07:43:08.493606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:08.490663 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" podUID="6c9e850f-e9fb-4aa6-ba8d-4113648338e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 07:43:09.487943 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:09.487902 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-b2nrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 07:43:09.488146 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:09.487970 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" podUID="6c9e850f-e9fb-4aa6-ba8d-4113648338e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 07:43:10.487771 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:10.487737 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-b2nrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 07:43:10.488273 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:10.487797 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" podUID="6c9e850f-e9fb-4aa6-ba8d-4113648338e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 07:43:11.488839 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:11.488792 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-b2nrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 07:43:11.489369 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:11.488866 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" podUID="6c9e850f-e9fb-4aa6-ba8d-4113648338e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 07:43:12.786414 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:12.786374 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" Oct 30 07:43:12.786950 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:12.786926 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" Oct 30 07:43:12.788072 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:12.788051 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-b2nrg" Oct 30 07:43:17.203201 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.203162 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff"] Oct 30 07:43:17.208403 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.208379 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:43:17.236631 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.236603 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff"] Oct 30 07:43:17.285080 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.285009 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2s5v\" (UniqueName: \"kubernetes.io/projected/69b69ee1-c4d2-4502-aea6-d9cd9716034e-kube-api-access-z2s5v\") pod \"maas-default-gateway-openshift-default-6dcf757899-s9mff\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:43:17.285406 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.285360 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-s9mff\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:43:17.285599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.285567 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-s9mff\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:43:17.285923 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.285896 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-s9mff\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:43:17.286103 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.286075 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-s9mff\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:43:17.286348 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.286315 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-s9mff\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:43:17.286535 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.286505 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-s9mff\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:43:17.288705 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.288675 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-s9mff\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:43:17.288956 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.288927 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-s9mff\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:43:17.390554 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.390506 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-s9mff\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:43:17.390554 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.390557 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-s9mff\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:43:17.390887 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.390634 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-s9mff\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:43:17.390887 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.390716 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-s9mff\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:43:17.390887 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.390745 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-s9mff\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:43:17.391073 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.390954 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2s5v\" (UniqueName: \"kubernetes.io/projected/69b69ee1-c4d2-4502-aea6-d9cd9716034e-kube-api-access-z2s5v\") pod \"maas-default-gateway-openshift-default-6dcf757899-s9mff\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:43:17.391073 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.391024 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-s9mff\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:43:17.391073 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.391025 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-s9mff\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:43:17.391073 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.390962 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-s9mff\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:43:17.391317 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.391102 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-s9mff\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:43:17.391317 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.391131 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-s9mff\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:43:17.391317 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.391302 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-s9mff\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:43:17.391485 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.391409 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-s9mff\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:43:17.391485 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.391438 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-s9mff\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:43:17.393216 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.393192 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-s9mff\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:43:17.393450 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.393432 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-s9mff\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:43:17.415958 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.415928 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-s9mff\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:43:17.416103 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.416084 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2s5v\" (UniqueName: \"kubernetes.io/projected/69b69ee1-c4d2-4502-aea6-d9cd9716034e-kube-api-access-z2s5v\") pod \"maas-default-gateway-openshift-default-6dcf757899-s9mff\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:43:17.523586 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.523553 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:43:17.856630 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:43:17.856560 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69b69ee1_c4d2_4502_aea6_d9cd9716034e.slice/crio-a838bc5e6454bb4765e39a213063b30bd7e71832ae1a7ec203774cf50b89846d WatchSource:0}: Error finding container a838bc5e6454bb4765e39a213063b30bd7e71832ae1a7ec203774cf50b89846d: Status 404 returned error can't find the container with id a838bc5e6454bb4765e39a213063b30bd7e71832ae1a7ec203774cf50b89846d Oct 30 07:43:17.856917 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.856896 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff"] Oct 30 07:43:17.859949 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.859903 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:43:17.860054 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.859997 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:43:17.860054 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:17.860045 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:43:18.539530 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:18.539493 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" event={"ID":"69b69ee1-c4d2-4502-aea6-d9cd9716034e","Type":"ContainerStarted","Data":"232990c27293634575c642b4d3af6a78380a4de5fe320a055a75b77f77aeac13"} Oct 30 07:43:18.540029 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:18.539555 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" event={"ID":"69b69ee1-c4d2-4502-aea6-d9cd9716034e","Type":"ContainerStarted","Data":"a838bc5e6454bb4765e39a213063b30bd7e71832ae1a7ec203774cf50b89846d"} Oct 30 07:43:18.606243 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:18.606189 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" podStartSLOduration=1.606172624 podStartE2EDuration="1.606172624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:43:18.601214394 +0000 UTC m=+2804.944262360" watchObservedRunningTime="2026-10-30 07:43:18.606172624 +0000 UTC m=+2804.949220588" Oct 30 07:43:19.527303 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:19.527267 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:43:19.539854 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:19.539812 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-s9mff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Oct 30 07:43:19.540481 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:19.540447 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" podUID="69b69ee1-c4d2-4502-aea6-d9cd9716034e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Oct 30 07:43:20.537603 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:20.526956 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-s9mff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Oct 30 07:43:20.537603 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:20.527018 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" podUID="69b69ee1-c4d2-4502-aea6-d9cd9716034e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Oct 30 07:43:21.543597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:21.538484 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-s9mff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Oct 30 07:43:21.543597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:21.538649 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" podUID="69b69ee1-c4d2-4502-aea6-d9cd9716034e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Oct 30 07:43:22.543597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:22.529337 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-s9mff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Oct 30 07:43:22.543597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:22.529403 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" podUID="69b69ee1-c4d2-4502-aea6-d9cd9716034e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Oct 30 07:43:23.061110 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:43:23.061073 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:43:23.535602 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:23.529971 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-s9mff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Oct 30 07:43:23.535602 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:23.530036 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" podUID="69b69ee1-c4d2-4502-aea6-d9cd9716034e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Oct 30 07:43:24.524143 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:24.524099 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-s9mff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Oct 30 07:43:24.524552 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:24.524168 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" podUID="69b69ee1-c4d2-4502-aea6-d9cd9716034e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Oct 30 07:43:25.527166 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:25.527130 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-s9mff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Oct 30 07:43:25.528346 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:25.528237 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" podUID="69b69ee1-c4d2-4502-aea6-d9cd9716034e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Oct 30 07:43:27.529616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:27.526467 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-s9mff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:43:27.529616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:27.526555 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" podUID="69b69ee1-c4d2-4502-aea6-d9cd9716034e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:43:27.577682 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:27.574008 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:43:27.587023 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:27.586935 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:43:27.630593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:27.626587 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:43:30.509594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:30.508664 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:43:30.647819 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:43:30.647714 4951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:43:53.046232 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:43:53.046190 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:44:02.364478 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:02.364431 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85"] Oct 30 07:44:02.370462 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:02.370437 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" Oct 30 07:44:02.396811 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:02.396780 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85"] Oct 30 07:44:02.435533 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:02.435495 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9287b67d-925d-46da-aa33-a5f47b3b7a1e-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-67v85\" (UID: \"9287b67d-925d-46da-aa33-a5f47b3b7a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" Oct 30 07:44:02.435777 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:02.435558 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9287b67d-925d-46da-aa33-a5f47b3b7a1e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-67v85\" (UID: \"9287b67d-925d-46da-aa33-a5f47b3b7a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" Oct 30 07:44:02.435777 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:02.435744 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9287b67d-925d-46da-aa33-a5f47b3b7a1e-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-67v85\" (UID: \"9287b67d-925d-46da-aa33-a5f47b3b7a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" Oct 30 07:44:02.435919 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:02.435791 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9287b67d-925d-46da-aa33-a5f47b3b7a1e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-67v85\" (UID: \"9287b67d-925d-46da-aa33-a5f47b3b7a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" Oct 30 07:44:02.435919 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:02.435828 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9287b67d-925d-46da-aa33-a5f47b3b7a1e-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-67v85\" (UID: \"9287b67d-925d-46da-aa33-a5f47b3b7a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" Oct 30 07:44:02.435919 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:02.435877 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9287b67d-925d-46da-aa33-a5f47b3b7a1e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-67v85\" (UID: \"9287b67d-925d-46da-aa33-a5f47b3b7a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" Oct 30 07:44:02.435919 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:02.435917 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfqld\" (UniqueName: \"kubernetes.io/projected/9287b67d-925d-46da-aa33-a5f47b3b7a1e-kube-api-access-cfqld\") pod \"maas-default-gateway-openshift-default-6dcf757899-67v85\" (UID: \"9287b67d-925d-46da-aa33-a5f47b3b7a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" Oct 30 07:44:02.436159 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:02.435957 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9287b67d-925d-46da-aa33-a5f47b3b7a1e-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-67v85\" (UID: \"9287b67d-925d-46da-aa33-a5f47b3b7a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" Oct 30 07:44:02.436159 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:02.435979 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9287b67d-925d-46da-aa33-a5f47b3b7a1e-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-67v85\" (UID: \"9287b67d-925d-46da-aa33-a5f47b3b7a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" Oct 30 07:44:02.537688 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:02.537659 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfqld\" (UniqueName: \"kubernetes.io/projected/9287b67d-925d-46da-aa33-a5f47b3b7a1e-kube-api-access-cfqld\") pod \"maas-default-gateway-openshift-default-6dcf757899-67v85\" (UID: \"9287b67d-925d-46da-aa33-a5f47b3b7a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" Oct 30 07:44:02.537851 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:02.537716 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9287b67d-925d-46da-aa33-a5f47b3b7a1e-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-67v85\" (UID: \"9287b67d-925d-46da-aa33-a5f47b3b7a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" Oct 30 07:44:02.537851 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:02.537739 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9287b67d-925d-46da-aa33-a5f47b3b7a1e-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-67v85\" (UID: \"9287b67d-925d-46da-aa33-a5f47b3b7a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" Oct 30 07:44:02.537851 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:02.537784 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9287b67d-925d-46da-aa33-a5f47b3b7a1e-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-67v85\" (UID: \"9287b67d-925d-46da-aa33-a5f47b3b7a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" Oct 30 07:44:02.537851 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:02.537812 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9287b67d-925d-46da-aa33-a5f47b3b7a1e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-67v85\" (UID: \"9287b67d-925d-46da-aa33-a5f47b3b7a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" Oct 30 07:44:02.538041 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:02.537931 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9287b67d-925d-46da-aa33-a5f47b3b7a1e-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-67v85\" (UID: \"9287b67d-925d-46da-aa33-a5f47b3b7a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" Oct 30 07:44:02.538041 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:02.537988 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9287b67d-925d-46da-aa33-a5f47b3b7a1e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-67v85\" (UID: \"9287b67d-925d-46da-aa33-a5f47b3b7a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" Oct 30 07:44:02.538175 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:02.538042 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9287b67d-925d-46da-aa33-a5f47b3b7a1e-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-67v85\" (UID: \"9287b67d-925d-46da-aa33-a5f47b3b7a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" Oct 30 07:44:02.538175 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:02.538125 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9287b67d-925d-46da-aa33-a5f47b3b7a1e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-67v85\" (UID: \"9287b67d-925d-46da-aa33-a5f47b3b7a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" Oct 30 07:44:02.538275 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:02.538168 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9287b67d-925d-46da-aa33-a5f47b3b7a1e-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-67v85\" (UID: \"9287b67d-925d-46da-aa33-a5f47b3b7a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" Oct 30 07:44:02.538275 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:02.538226 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9287b67d-925d-46da-aa33-a5f47b3b7a1e-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-67v85\" (UID: \"9287b67d-925d-46da-aa33-a5f47b3b7a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" Oct 30 07:44:02.538364 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:02.538345 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9287b67d-925d-46da-aa33-a5f47b3b7a1e-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-67v85\" (UID: \"9287b67d-925d-46da-aa33-a5f47b3b7a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" Oct 30 07:44:02.538454 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:02.538431 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9287b67d-925d-46da-aa33-a5f47b3b7a1e-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-67v85\" (UID: \"9287b67d-925d-46da-aa33-a5f47b3b7a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" Oct 30 07:44:02.538685 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:02.538660 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9287b67d-925d-46da-aa33-a5f47b3b7a1e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-67v85\" (UID: \"9287b67d-925d-46da-aa33-a5f47b3b7a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" Oct 30 07:44:02.540395 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:02.540374 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9287b67d-925d-46da-aa33-a5f47b3b7a1e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-67v85\" (UID: \"9287b67d-925d-46da-aa33-a5f47b3b7a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" Oct 30 07:44:02.540502 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:02.540484 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9287b67d-925d-46da-aa33-a5f47b3b7a1e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-67v85\" (UID: \"9287b67d-925d-46da-aa33-a5f47b3b7a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" Oct 30 07:44:02.562330 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:02.562302 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9287b67d-925d-46da-aa33-a5f47b3b7a1e-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-67v85\" (UID: \"9287b67d-925d-46da-aa33-a5f47b3b7a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" Oct 30 07:44:02.562444 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:02.562390 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfqld\" (UniqueName: \"kubernetes.io/projected/9287b67d-925d-46da-aa33-a5f47b3b7a1e-kube-api-access-cfqld\") pod \"maas-default-gateway-openshift-default-6dcf757899-67v85\" (UID: \"9287b67d-925d-46da-aa33-a5f47b3b7a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" Oct 30 07:44:02.687892 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:02.687798 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" Oct 30 07:44:03.063966 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:03.063931 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85"] Oct 30 07:44:03.065590 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:44:03.065382 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9287b67d_925d_46da_aa33_a5f47b3b7a1e.slice/crio-3f426871d7e2940df05d472fa904176cbe7dd661c23f73dc180b3846f346e4f1 WatchSource:0}: Error finding container 3f426871d7e2940df05d472fa904176cbe7dd661c23f73dc180b3846f346e4f1: Status 404 returned error can't find the container with id 3f426871d7e2940df05d472fa904176cbe7dd661c23f73dc180b3846f346e4f1 Oct 30 07:44:03.086966 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:03.086284 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:44:03.086966 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:03.086396 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:44:03.086966 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:03.086445 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:44:03.351300 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:03.351235 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" event={"ID":"9287b67d-925d-46da-aa33-a5f47b3b7a1e","Type":"ContainerStarted","Data":"3f426871d7e2940df05d472fa904176cbe7dd661c23f73dc180b3846f346e4f1"} Oct 30 07:44:04.362007 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:04.361966 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" event={"ID":"9287b67d-925d-46da-aa33-a5f47b3b7a1e","Type":"ContainerStarted","Data":"05cc3860169e9ed827c506c79880955b9471298de0c1c195cc0c707008339a84"} Oct 30 07:44:04.450488 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:04.448532 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" podStartSLOduration=2.448515269 podStartE2EDuration="2.448515269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:44:04.447776516 +0000 UTC m=+2850.790824482" watchObservedRunningTime="2026-10-30 07:44:04.448515269 +0000 UTC m=+2850.791563234" Oct 30 07:44:04.688160 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:04.688081 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" Oct 30 07:44:04.690281 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:04.690254 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-67v85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 30 07:44:04.690422 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:04.690304 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" podUID="9287b67d-925d-46da-aa33-a5f47b3b7a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 30 07:44:05.688798 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:05.688760 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-67v85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 30 07:44:05.689225 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:05.688813 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" podUID="9287b67d-925d-46da-aa33-a5f47b3b7a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 30 07:44:06.688330 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:06.688291 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-67v85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 30 07:44:06.688507 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:06.688350 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" podUID="9287b67d-925d-46da-aa33-a5f47b3b7a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 30 07:44:07.688206 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:07.688172 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-67v85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 30 07:44:07.688660 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:07.688247 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" podUID="9287b67d-925d-46da-aa33-a5f47b3b7a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 30 07:44:08.688542 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:08.688505 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-67v85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 30 07:44:08.689061 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:08.688567 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" podUID="9287b67d-925d-46da-aa33-a5f47b3b7a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 30 07:44:09.689189 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:09.689149 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-67v85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 30 07:44:09.689690 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:09.689209 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" podUID="9287b67d-925d-46da-aa33-a5f47b3b7a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 30 07:44:10.455594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:10.454024 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd"] Oct 30 07:44:10.700997 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:10.700892 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-67v85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 30 07:44:10.701745 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:10.701550 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" podUID="9287b67d-925d-46da-aa33-a5f47b3b7a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 30 07:44:10.738988 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:10.738902 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd"] Oct 30 07:44:10.739461 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:10.739425 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:44:10.772812 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:10.772780 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d5f604-openshift-default-dockercfg-mkjm6\"" Oct 30 07:44:10.902080 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:10.902035 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-workload-socket\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:44:10.902288 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:10.902094 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istio-envoy\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:44:10.902288 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:10.902178 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istiod-ca-cert\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:44:10.902288 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:10.902269 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-credential-socket\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:44:10.902922 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:10.902334 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istio-data\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:44:10.902922 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:10.902431 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istio-token\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:44:10.902922 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:10.902466 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istio-podinfo\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:44:10.902922 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:10.902544 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-workload-certs\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:44:10.902922 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:10.902616 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9ht5\" (UniqueName: \"kubernetes.io/projected/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-kube-api-access-c9ht5\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:44:11.003812 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:11.003781 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istio-podinfo\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:44:11.004016 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:11.003828 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-workload-certs\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:44:11.004182 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:11.004148 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9ht5\" (UniqueName: \"kubernetes.io/projected/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-kube-api-access-c9ht5\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:44:11.004563 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:11.004223 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-workload-certs\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:44:11.004827 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:11.004677 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-workload-socket\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:44:11.004963 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:11.004941 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-workload-socket\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:44:11.005108 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:11.005012 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istio-envoy\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:44:11.005212 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:11.005125 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istiod-ca-cert\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:44:11.005212 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:11.005175 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-credential-socket\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:44:11.005346 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:11.005243 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istio-data\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:44:11.005346 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:11.005316 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istio-token\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:44:11.005972 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:11.005946 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istiod-ca-cert\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:44:11.006455 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:11.006432 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-credential-socket\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:44:11.006638 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:11.006482 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istio-data\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:44:11.008603 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:11.008562 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istio-envoy\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:44:11.009054 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:11.009021 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istio-podinfo\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:44:11.029379 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:11.029352 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9ht5\" (UniqueName: \"kubernetes.io/projected/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-kube-api-access-c9ht5\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:44:11.029921 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:11.029896 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istio-token\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:44:11.062559 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:11.062290 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:44:11.411909 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:44:11.411874 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac54b4ca_6422_4b37_b14c_7a0b8e3657cc.slice/crio-5f26d507e89a728fc1c32bde27c8584c8ae28939acdc178a9d865b4e40a9c536 WatchSource:0}: Error finding container 5f26d507e89a728fc1c32bde27c8584c8ae28939acdc178a9d865b4e40a9c536: Status 404 returned error can't find the container with id 5f26d507e89a728fc1c32bde27c8584c8ae28939acdc178a9d865b4e40a9c536 Oct 30 07:44:11.413417 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:11.413380 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd"] Oct 30 07:44:11.415396 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:11.415356 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:44:11.415489 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:11.415444 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:44:11.415489 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:11.415478 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:44:11.731101 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:11.730998 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" event={"ID":"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc","Type":"ContainerStarted","Data":"139501cb783671fd2c36975c9c1a71091246a217a8862ca1ca657566557ec956"} Oct 30 07:44:11.731101 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:11.731072 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" event={"ID":"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc","Type":"ContainerStarted","Data":"5f26d507e89a728fc1c32bde27c8584c8ae28939acdc178a9d865b4e40a9c536"} Oct 30 07:44:11.844054 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:11.843988 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" podStartSLOduration=1.8439675530000001 podStartE2EDuration="1.843967553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:44:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:44:11.7811979 +0000 UTC m=+2858.124245885" watchObservedRunningTime="2026-10-30 07:44:11.843967553 +0000 UTC m=+2858.187015527" Oct 30 07:44:12.065876 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:12.065840 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:44:12.071935 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:12.071913 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:44:12.667116 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:12.667082 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" Oct 30 07:44:12.667935 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:12.667906 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" Oct 30 07:44:12.668736 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:12.668714 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-67v85" Oct 30 07:44:12.740213 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:12.740178 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:44:12.741417 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:12.741393 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:44:23.025395 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:44:23.025351 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:44:45.135971 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:45.135925 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4"] Oct 30 07:44:45.142403 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:45.142365 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:44:45.165449 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:45.165409 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4"] Oct 30 07:44:45.176641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:45.172874 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-fae1d2-openshift-default-dockercfg-qlxwn\"" Oct 30 07:44:45.251340 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:45.251303 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-credential-socket\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:44:45.251543 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:45.251348 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7771d5de-79de-4e6d-a0e8-acbacaf902af-istio-token\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:44:45.251543 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:45.251487 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7771d5de-79de-4e6d-a0e8-acbacaf902af-istiod-ca-cert\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:44:45.251696 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:45.251680 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-istio-envoy\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:44:45.251756 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:45.251716 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wdqk\" (UniqueName: \"kubernetes.io/projected/7771d5de-79de-4e6d-a0e8-acbacaf902af-kube-api-access-2wdqk\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:44:45.251839 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:45.251821 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-workload-certs\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:44:45.251895 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:45.251883 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-workload-socket\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:44:45.252005 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:45.251992 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-istio-data\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:44:45.252064 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:45.252052 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7771d5de-79de-4e6d-a0e8-acbacaf902af-istio-podinfo\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:44:45.357696 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:45.357652 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-istio-envoy\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:44:45.358504 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:45.358479 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wdqk\" (UniqueName: \"kubernetes.io/projected/7771d5de-79de-4e6d-a0e8-acbacaf902af-kube-api-access-2wdqk\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:44:45.358717 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:45.358700 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-workload-certs\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:44:45.358922 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:45.358906 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-workload-socket\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:44:45.359082 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:45.359067 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-istio-data\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:44:45.359251 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:45.359238 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7771d5de-79de-4e6d-a0e8-acbacaf902af-istio-podinfo\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:44:45.359397 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:45.359385 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-credential-socket\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:44:45.359501 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:45.359488 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7771d5de-79de-4e6d-a0e8-acbacaf902af-istio-token\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:44:45.359680 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:45.359665 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7771d5de-79de-4e6d-a0e8-acbacaf902af-istiod-ca-cert\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:44:45.362860 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:45.360785 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-workload-socket\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:44:45.362860 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:45.361077 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-istio-data\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:44:45.362860 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:45.362120 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7771d5de-79de-4e6d-a0e8-acbacaf902af-istiod-ca-cert\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:44:45.362860 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:45.362449 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-workload-certs\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:44:45.363553 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:45.363516 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-credential-socket\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:44:45.425296 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:45.404893 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-istio-envoy\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:44:45.425535 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:45.422198 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7771d5de-79de-4e6d-a0e8-acbacaf902af-istio-token\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:44:45.429645 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:45.424646 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wdqk\" (UniqueName: \"kubernetes.io/projected/7771d5de-79de-4e6d-a0e8-acbacaf902af-kube-api-access-2wdqk\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:44:45.458026 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:45.425169 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7771d5de-79de-4e6d-a0e8-acbacaf902af-istio-podinfo\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:44:45.470127 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:45.469871 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:44:46.020683 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:44:46.020647 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7771d5de_79de_4e6d_a0e8_acbacaf902af.slice/crio-fed9beabd50518f8bf56161bb0fcf83561aea32c3be5e4633eca84ea73fb54d6 WatchSource:0}: Error finding container fed9beabd50518f8bf56161bb0fcf83561aea32c3be5e4633eca84ea73fb54d6: Status 404 returned error can't find the container with id fed9beabd50518f8bf56161bb0fcf83561aea32c3be5e4633eca84ea73fb54d6 Oct 30 07:44:46.022015 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:46.021987 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4"] Oct 30 07:44:46.024156 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:46.024117 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:44:46.024275 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:46.024208 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:44:46.024275 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:46.024255 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:44:46.187638 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:46.187600 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" event={"ID":"7771d5de-79de-4e6d-a0e8-acbacaf902af","Type":"ContainerStarted","Data":"e6a584ae28bee7ee12fa124066766c0857e3af029a91b49eec1f1739d75fba0f"} Oct 30 07:44:46.188119 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:46.187661 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" event={"ID":"7771d5de-79de-4e6d-a0e8-acbacaf902af","Type":"ContainerStarted","Data":"fed9beabd50518f8bf56161bb0fcf83561aea32c3be5e4633eca84ea73fb54d6"} Oct 30 07:44:46.313142 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:46.313024 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" podStartSLOduration=1.313009731 podStartE2EDuration="1.313009731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:44:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:44:46.307118009 +0000 UTC m=+2892.650165977" watchObservedRunningTime="2026-10-30 07:44:46.313009731 +0000 UTC m=+2892.656057744" Oct 30 07:44:46.493595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:46.491704 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:44:47.497451 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:47.497419 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:44:48.204190 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:48.204159 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:44:48.205075 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:44:48.205051 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:44:53.024848 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:44:53.024811 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:45:00.557597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:00.556632 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7"] Oct 30 07:45:00.642254 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:00.640849 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7" Oct 30 07:45:00.658674 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:00.658647 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg"] Oct 30 07:45:00.665051 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:00.665022 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7"] Oct 30 07:45:00.665241 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:00.665140 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg" Oct 30 07:45:00.668416 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:00.668246 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg"] Oct 30 07:45:00.671037 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:00.671013 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:45:00.718782 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:00.718738 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kjnb\" (UniqueName: \"kubernetes.io/projected/a859e7f4-ca02-45e7-b64c-9b71b4d88311-kube-api-access-7kjnb\") pod \"maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7\" (UID: \"a859e7f4-ca02-45e7-b64c-9b71b4d88311\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7" Oct 30 07:45:00.718969 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:00.718821 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7cf2\" (UniqueName: \"kubernetes.io/projected/bb22b95a-6a2c-4727-80a1-258347acd504-kube-api-access-j7cf2\") pod \"maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg\" (UID: \"bb22b95a-6a2c-4727-80a1-258347acd504\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg" Oct 30 07:45:00.718969 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:00.718945 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a859e7f4-ca02-45e7-b64c-9b71b4d88311-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7\" (UID: \"a859e7f4-ca02-45e7-b64c-9b71b4d88311\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7" Oct 30 07:45:00.719279 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:00.719261 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb22b95a-6a2c-4727-80a1-258347acd504-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg\" (UID: \"bb22b95a-6a2c-4727-80a1-258347acd504\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg" Oct 30 07:45:00.821416 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:00.821319 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb22b95a-6a2c-4727-80a1-258347acd504-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg\" (UID: \"bb22b95a-6a2c-4727-80a1-258347acd504\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg" Oct 30 07:45:00.821652 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:00.821464 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kjnb\" (UniqueName: \"kubernetes.io/projected/a859e7f4-ca02-45e7-b64c-9b71b4d88311-kube-api-access-7kjnb\") pod \"maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7\" (UID: \"a859e7f4-ca02-45e7-b64c-9b71b4d88311\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7" Oct 30 07:45:00.821652 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:00.821534 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7cf2\" (UniqueName: \"kubernetes.io/projected/bb22b95a-6a2c-4727-80a1-258347acd504-kube-api-access-j7cf2\") pod \"maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg\" (UID: \"bb22b95a-6a2c-4727-80a1-258347acd504\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg" Oct 30 07:45:00.821961 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:00.821927 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a859e7f4-ca02-45e7-b64c-9b71b4d88311-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7\" (UID: \"a859e7f4-ca02-45e7-b64c-9b71b4d88311\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7" Oct 30 07:45:00.822525 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:00.822498 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb22b95a-6a2c-4727-80a1-258347acd504-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg\" (UID: \"bb22b95a-6a2c-4727-80a1-258347acd504\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg" Oct 30 07:45:00.822801 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:00.822778 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a859e7f4-ca02-45e7-b64c-9b71b4d88311-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7\" (UID: \"a859e7f4-ca02-45e7-b64c-9b71b4d88311\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7" Oct 30 07:45:00.853206 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:00.853163 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7cf2\" (UniqueName: \"kubernetes.io/projected/bb22b95a-6a2c-4727-80a1-258347acd504-kube-api-access-j7cf2\") pod \"maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg\" (UID: \"bb22b95a-6a2c-4727-80a1-258347acd504\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg" Oct 30 07:45:00.856647 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:00.855001 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kjnb\" (UniqueName: \"kubernetes.io/projected/a859e7f4-ca02-45e7-b64c-9b71b4d88311-kube-api-access-7kjnb\") pod \"maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7\" (UID: \"a859e7f4-ca02-45e7-b64c-9b71b4d88311\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7" Oct 30 07:45:00.962679 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:00.962631 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7" Oct 30 07:45:00.988738 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:00.988676 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg" Oct 30 07:45:01.642277 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:45:01.642240 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb22b95a_6a2c_4727_80a1_258347acd504.slice/crio-1989828aa7bb267cd9b23e16d2957b31309cafef1677499286c68bef8c1e089e WatchSource:0}: Error finding container 1989828aa7bb267cd9b23e16d2957b31309cafef1677499286c68bef8c1e089e: Status 404 returned error can't find the container with id 1989828aa7bb267cd9b23e16d2957b31309cafef1677499286c68bef8c1e089e Oct 30 07:45:01.642917 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:01.642885 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg"] Oct 30 07:45:01.670850 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:45:01.670792 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda859e7f4_ca02_45e7_b64c_9b71b4d88311.slice/crio-710e0bb9112208f50a76d8c7cf764b738dd38daae000c6737dd85d843890691b WatchSource:0}: Error finding container 710e0bb9112208f50a76d8c7cf764b738dd38daae000c6737dd85d843890691b: Status 404 returned error can't find the container with id 710e0bb9112208f50a76d8c7cf764b738dd38daae000c6737dd85d843890691b Oct 30 07:45:01.671868 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:01.671836 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7"] Oct 30 07:45:02.426636 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:02.425968 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7" event={"ID":"a859e7f4-ca02-45e7-b64c-9b71b4d88311","Type":"ContainerStarted","Data":"ce950125dce76fe4c0903b0dfe345f4ab4fd82fa50821454a939df26ec13c718"} Oct 30 07:45:02.426636 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:02.426034 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7" event={"ID":"a859e7f4-ca02-45e7-b64c-9b71b4d88311","Type":"ContainerStarted","Data":"710e0bb9112208f50a76d8c7cf764b738dd38daae000c6737dd85d843890691b"} Oct 30 07:45:02.428314 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:02.428277 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg" event={"ID":"bb22b95a-6a2c-4727-80a1-258347acd504","Type":"ContainerStarted","Data":"e0ab6b523e22b4747b5f784c9724e9d32302cdf83b11c070cf97d203f7e3df37"} Oct 30 07:45:02.428472 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:02.428334 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg" event={"ID":"bb22b95a-6a2c-4727-80a1-258347acd504","Type":"ContainerStarted","Data":"1989828aa7bb267cd9b23e16d2957b31309cafef1677499286c68bef8c1e089e"} Oct 30 07:45:02.487888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:02.487757 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7" podStartSLOduration=2.487736248 podStartE2EDuration="2.487736248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:02.478130754 +0000 UTC m=+2908.821178733" watchObservedRunningTime="2026-10-30 07:45:02.487736248 +0000 UTC m=+2908.830784214" Oct 30 07:45:02.525853 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:02.525790 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg" podStartSLOduration=2.525770226 podStartE2EDuration="2.525770226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:02.522777856 +0000 UTC m=+2908.865825821" watchObservedRunningTime="2026-10-30 07:45:02.525770226 +0000 UTC m=+2908.868818205" Oct 30 07:45:03.443430 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:03.443097 4951 generic.go:356] "Generic (PLEG): container finished" podID="a859e7f4-ca02-45e7-b64c-9b71b4d88311" containerID="ce950125dce76fe4c0903b0dfe345f4ab4fd82fa50821454a939df26ec13c718" exitCode=60 Oct 30 07:45:03.443430 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:03.443184 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7" event={"ID":"a859e7f4-ca02-45e7-b64c-9b71b4d88311","Type":"ContainerDied","Data":"ce950125dce76fe4c0903b0dfe345f4ab4fd82fa50821454a939df26ec13c718"} Oct 30 07:45:03.444756 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:03.444338 4951 scope.go:122] "RemoveContainer" containerID="ce950125dce76fe4c0903b0dfe345f4ab4fd82fa50821454a939df26ec13c718" Oct 30 07:45:03.446103 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:03.445947 4951 generic.go:356] "Generic (PLEG): container finished" podID="bb22b95a-6a2c-4727-80a1-258347acd504" containerID="e0ab6b523e22b4747b5f784c9724e9d32302cdf83b11c070cf97d203f7e3df37" exitCode=60 Oct 30 07:45:03.446196 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:03.446168 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg" event={"ID":"bb22b95a-6a2c-4727-80a1-258347acd504","Type":"ContainerDied","Data":"e0ab6b523e22b4747b5f784c9724e9d32302cdf83b11c070cf97d203f7e3df37"} Oct 30 07:45:03.446550 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:03.446535 4951 scope.go:122] "RemoveContainer" containerID="e0ab6b523e22b4747b5f784c9724e9d32302cdf83b11c070cf97d203f7e3df37" Oct 30 07:45:04.537715 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:04.530306 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7" event={"ID":"a859e7f4-ca02-45e7-b64c-9b71b4d88311","Type":"ContainerStarted","Data":"1923b339c17195395b0ae856219ff42ba1f2011fdd2b7633c28cd86ac4207b1a"} Oct 30 07:45:04.553343 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:04.553305 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg" event={"ID":"bb22b95a-6a2c-4727-80a1-258347acd504","Type":"ContainerStarted","Data":"f1836d7ace826eb8c8bfa0ef5313adeaf612f82d40caad0d43bc4ab36b593549"} Oct 30 07:45:04.969662 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:45:04.969609 4951 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda859e7f4_ca02_45e7_b64c_9b71b4d88311.slice/crio-conmon-1923b339c17195395b0ae856219ff42ba1f2011fdd2b7633c28cd86ac4207b1a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda859e7f4_ca02_45e7_b64c_9b71b4d88311.slice/crio-1923b339c17195395b0ae856219ff42ba1f2011fdd2b7633c28cd86ac4207b1a.scope\": RecentStats: unable to find data in memory cache]" Oct 30 07:45:04.972357 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:45:04.969796 4951 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda859e7f4_ca02_45e7_b64c_9b71b4d88311.slice/crio-1923b339c17195395b0ae856219ff42ba1f2011fdd2b7633c28cd86ac4207b1a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb22b95a_6a2c_4727_80a1_258347acd504.slice/crio-f1836d7ace826eb8c8bfa0ef5313adeaf612f82d40caad0d43bc4ab36b593549.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda859e7f4_ca02_45e7_b64c_9b71b4d88311.slice/crio-conmon-1923b339c17195395b0ae856219ff42ba1f2011fdd2b7633c28cd86ac4207b1a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb22b95a_6a2c_4727_80a1_258347acd504.slice/crio-conmon-f1836d7ace826eb8c8bfa0ef5313adeaf612f82d40caad0d43bc4ab36b593549.scope\": RecentStats: unable to find data in memory cache]" Oct 30 07:45:04.979884 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:04.979833 4951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:45:05.567661 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:05.567620 4951 generic.go:356] "Generic (PLEG): container finished" podID="a859e7f4-ca02-45e7-b64c-9b71b4d88311" containerID="1923b339c17195395b0ae856219ff42ba1f2011fdd2b7633c28cd86ac4207b1a" exitCode=60 Oct 30 07:45:05.568364 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:05.568153 4951 scope.go:122] "RemoveContainer" containerID="1923b339c17195395b0ae856219ff42ba1f2011fdd2b7633c28cd86ac4207b1a" Oct 30 07:45:05.568497 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:05.568244 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7" event={"ID":"a859e7f4-ca02-45e7-b64c-9b71b4d88311","Type":"ContainerDied","Data":"1923b339c17195395b0ae856219ff42ba1f2011fdd2b7633c28cd86ac4207b1a"} Oct 30 07:45:05.568587 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:05.568557 4951 scope.go:122] "RemoveContainer" containerID="ce950125dce76fe4c0903b0dfe345f4ab4fd82fa50821454a939df26ec13c718" Oct 30 07:45:05.568873 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:45:05.568849 4951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7_odh-ai-gateway-infra(a859e7f4-ca02-45e7-b64c-9b71b4d88311)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7" podUID="a859e7f4-ca02-45e7-b64c-9b71b4d88311" Oct 30 07:45:05.574709 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:05.573530 4951 generic.go:356] "Generic (PLEG): container finished" podID="bb22b95a-6a2c-4727-80a1-258347acd504" containerID="f1836d7ace826eb8c8bfa0ef5313adeaf612f82d40caad0d43bc4ab36b593549" exitCode=60 Oct 30 07:45:05.574709 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:05.573658 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg" event={"ID":"bb22b95a-6a2c-4727-80a1-258347acd504","Type":"ContainerDied","Data":"f1836d7ace826eb8c8bfa0ef5313adeaf612f82d40caad0d43bc4ab36b593549"} Oct 30 07:45:05.574709 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:05.574127 4951 scope.go:122] "RemoveContainer" containerID="f1836d7ace826eb8c8bfa0ef5313adeaf612f82d40caad0d43bc4ab36b593549" Oct 30 07:45:05.574709 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:45:05.574511 4951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg_odh-ai-gateway-infra(bb22b95a-6a2c-4727-80a1-258347acd504)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg" podUID="bb22b95a-6a2c-4727-80a1-258347acd504" Oct 30 07:45:05.589209 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:05.589186 4951 scope.go:122] "RemoveContainer" containerID="e0ab6b523e22b4747b5f784c9724e9d32302cdf83b11c070cf97d203f7e3df37" Oct 30 07:45:06.600622 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:06.596878 4951 scope.go:122] "RemoveContainer" containerID="1923b339c17195395b0ae856219ff42ba1f2011fdd2b7633c28cd86ac4207b1a" Oct 30 07:45:06.600622 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:45:06.597439 4951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7_odh-ai-gateway-infra(a859e7f4-ca02-45e7-b64c-9b71b4d88311)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7" podUID="a859e7f4-ca02-45e7-b64c-9b71b4d88311" Oct 30 07:45:06.604351 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:06.601637 4951 scope.go:122] "RemoveContainer" containerID="f1836d7ace826eb8c8bfa0ef5313adeaf612f82d40caad0d43bc4ab36b593549" Oct 30 07:45:06.604351 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:45:06.602107 4951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg_odh-ai-gateway-infra(bb22b95a-6a2c-4727-80a1-258347acd504)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg" podUID="bb22b95a-6a2c-4727-80a1-258347acd504" Oct 30 07:45:16.765216 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:16.764832 4951 scope.go:122] "RemoveContainer" containerID="f1836d7ace826eb8c8bfa0ef5313adeaf612f82d40caad0d43bc4ab36b593549" Oct 30 07:45:17.732396 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:17.732364 4951 generic.go:356] "Generic (PLEG): container finished" podID="bb22b95a-6a2c-4727-80a1-258347acd504" containerID="901ab9005a6eb0daa85689c7e04a56391a3225a60626dd04d04658ab8e2296fc" exitCode=60 Oct 30 07:45:17.732623 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:17.732432 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg" event={"ID":"bb22b95a-6a2c-4727-80a1-258347acd504","Type":"ContainerDied","Data":"901ab9005a6eb0daa85689c7e04a56391a3225a60626dd04d04658ab8e2296fc"} Oct 30 07:45:17.732623 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:17.732523 4951 scope.go:122] "RemoveContainer" containerID="f1836d7ace826eb8c8bfa0ef5313adeaf612f82d40caad0d43bc4ab36b593549" Oct 30 07:45:17.733062 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:17.733034 4951 scope.go:122] "RemoveContainer" containerID="901ab9005a6eb0daa85689c7e04a56391a3225a60626dd04d04658ab8e2296fc" Oct 30 07:45:17.733508 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:45:17.733477 4951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg_odh-ai-gateway-infra(bb22b95a-6a2c-4727-80a1-258347acd504)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg" podUID="bb22b95a-6a2c-4727-80a1-258347acd504" Oct 30 07:45:17.764721 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:17.764693 4951 scope.go:122] "RemoveContainer" containerID="1923b339c17195395b0ae856219ff42ba1f2011fdd2b7633c28cd86ac4207b1a" Oct 30 07:45:18.746158 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:18.746121 4951 generic.go:356] "Generic (PLEG): container finished" podID="a859e7f4-ca02-45e7-b64c-9b71b4d88311" containerID="a66ff2f627e5407e1b194b7f017b2ac03aff6299fdc6174456ff1d52c3ff7b93" exitCode=60 Oct 30 07:45:18.746678 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:18.746189 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7" event={"ID":"a859e7f4-ca02-45e7-b64c-9b71b4d88311","Type":"ContainerDied","Data":"a66ff2f627e5407e1b194b7f017b2ac03aff6299fdc6174456ff1d52c3ff7b93"} Oct 30 07:45:18.746678 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:18.746258 4951 scope.go:122] "RemoveContainer" containerID="1923b339c17195395b0ae856219ff42ba1f2011fdd2b7633c28cd86ac4207b1a" Oct 30 07:45:18.746801 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:18.746771 4951 scope.go:122] "RemoveContainer" containerID="a66ff2f627e5407e1b194b7f017b2ac03aff6299fdc6174456ff1d52c3ff7b93" Oct 30 07:45:18.747241 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:45:18.747210 4951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7_odh-ai-gateway-infra(a859e7f4-ca02-45e7-b64c-9b71b4d88311)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7" podUID="a859e7f4-ca02-45e7-b64c-9b71b4d88311" Oct 30 07:45:18.806433 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:18.806402 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg"] Oct 30 07:45:19.015949 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:19.015915 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg" Oct 30 07:45:19.075722 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:19.075684 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb22b95a-6a2c-4727-80a1-258347acd504-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb22b95a-6a2c-4727-80a1-258347acd504-maas-api-service-ca\") pod \"bb22b95a-6a2c-4727-80a1-258347acd504\" (UID: \"bb22b95a-6a2c-4727-80a1-258347acd504\") " Oct 30 07:45:19.075926 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:19.075807 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb22b95a-6a2c-4727-80a1-258347acd504-kube-api-access-j7cf2\" (UniqueName: \"kubernetes.io/projected/bb22b95a-6a2c-4727-80a1-258347acd504-kube-api-access-j7cf2\") pod \"bb22b95a-6a2c-4727-80a1-258347acd504\" (UID: \"bb22b95a-6a2c-4727-80a1-258347acd504\") " Oct 30 07:45:19.076960 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:19.076914 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb22b95a-6a2c-4727-80a1-258347acd504-maas-api-service-ca" pod "bb22b95a-6a2c-4727-80a1-258347acd504" (UID: "bb22b95a-6a2c-4727-80a1-258347acd504"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:45:19.078646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:19.078616 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb22b95a-6a2c-4727-80a1-258347acd504-kube-api-access-j7cf2" pod "bb22b95a-6a2c-4727-80a1-258347acd504" (UID: "bb22b95a-6a2c-4727-80a1-258347acd504"). InnerVolumeSpecName "kube-api-access-j7cf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:45:19.178359 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:19.178317 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb22b95a-6a2c-4727-80a1-258347acd504-maas-api-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:45:19.178359 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:19.178360 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7cf2\" (UniqueName: \"kubernetes.io/projected/bb22b95a-6a2c-4727-80a1-258347acd504-kube-api-access-j7cf2\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:45:19.760551 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:19.760517 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg" Oct 30 07:45:19.760551 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:19.760533 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg" event={"ID":"bb22b95a-6a2c-4727-80a1-258347acd504","Type":"ContainerDied","Data":"1989828aa7bb267cd9b23e16d2957b31309cafef1677499286c68bef8c1e089e"} Oct 30 07:45:19.761112 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:19.760647 4951 scope.go:122] "RemoveContainer" containerID="901ab9005a6eb0daa85689c7e04a56391a3225a60626dd04d04658ab8e2296fc" Oct 30 07:45:19.813526 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:19.813491 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7"] Oct 30 07:45:19.844728 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:19.844693 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg"] Oct 30 07:45:19.851430 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:19.851388 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889105-tjvgg"] Oct 30 07:45:20.089069 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:20.089032 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7" Oct 30 07:45:20.159673 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:20.159639 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a859e7f4-ca02-45e7-b64c-9b71b4d88311-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a859e7f4-ca02-45e7-b64c-9b71b4d88311-maas-api-service-ca\") pod \"a859e7f4-ca02-45e7-b64c-9b71b4d88311\" (UID: \"a859e7f4-ca02-45e7-b64c-9b71b4d88311\") " Oct 30 07:45:20.159899 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:20.159873 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a859e7f4-ca02-45e7-b64c-9b71b4d88311-kube-api-access-7kjnb\" (UniqueName: \"kubernetes.io/projected/a859e7f4-ca02-45e7-b64c-9b71b4d88311-kube-api-access-7kjnb\") pod \"a859e7f4-ca02-45e7-b64c-9b71b4d88311\" (UID: \"a859e7f4-ca02-45e7-b64c-9b71b4d88311\") " Oct 30 07:45:20.160619 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:20.160404 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a859e7f4-ca02-45e7-b64c-9b71b4d88311-maas-api-service-ca" pod "a859e7f4-ca02-45e7-b64c-9b71b4d88311" (UID: "a859e7f4-ca02-45e7-b64c-9b71b4d88311"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:45:20.164638 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:20.164608 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a859e7f4-ca02-45e7-b64c-9b71b4d88311-kube-api-access-7kjnb" pod "a859e7f4-ca02-45e7-b64c-9b71b4d88311" (UID: "a859e7f4-ca02-45e7-b64c-9b71b4d88311"). InnerVolumeSpecName "kube-api-access-7kjnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:45:20.262227 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:20.262196 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kjnb\" (UniqueName: \"kubernetes.io/projected/a859e7f4-ca02-45e7-b64c-9b71b4d88311-kube-api-access-7kjnb\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:45:20.262227 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:20.262226 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a859e7f4-ca02-45e7-b64c-9b71b4d88311-maas-api-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:45:20.775494 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:20.775448 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7" event={"ID":"a859e7f4-ca02-45e7-b64c-9b71b4d88311","Type":"ContainerDied","Data":"710e0bb9112208f50a76d8c7cf764b738dd38daae000c6737dd85d843890691b"} Oct 30 07:45:20.776393 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:20.775528 4951 scope.go:122] "RemoveContainer" containerID="a66ff2f627e5407e1b194b7f017b2ac03aff6299fdc6174456ff1d52c3ff7b93" Oct 30 07:45:20.776393 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:20.775456 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7" Oct 30 07:45:20.836425 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:20.836386 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7"] Oct 30 07:45:20.840941 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:20.840907 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889105-w2fs7"] Oct 30 07:45:21.777078 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:21.777030 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a859e7f4-ca02-45e7-b64c-9b71b4d88311" path="/var/lib/kubelet/pods/a859e7f4-ca02-45e7-b64c-9b71b4d88311/volumes" Oct 30 07:45:21.777806 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:21.777783 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb22b95a-6a2c-4727-80a1-258347acd504" path="/var/lib/kubelet/pods/bb22b95a-6a2c-4727-80a1-258347acd504/volumes" Oct 30 07:45:23.028024 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:45:23.027974 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:45:50.068526 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:50.068486 4951 generic.go:356] "Generic (PLEG): container finished" podID="ac54b4ca-6422-4b37-b14c-7a0b8e3657cc" containerID="139501cb783671fd2c36975c9c1a71091246a217a8862ca1ca657566557ec956" exitCode=0 Oct 30 07:45:50.069068 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:50.068560 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" event={"ID":"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc","Type":"ContainerDied","Data":"139501cb783671fd2c36975c9c1a71091246a217a8862ca1ca657566557ec956"} Oct 30 07:45:50.069155 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:50.069139 4951 scope.go:122] "RemoveContainer" containerID="139501cb783671fd2c36975c9c1a71091246a217a8862ca1ca657566557ec956" Oct 30 07:45:50.071415 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:50.071377 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:45:50.071534 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:50.071474 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:45:50.071626 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:50.071534 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:45:51.084839 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:51.084792 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" event={"ID":"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc","Type":"ContainerStarted","Data":"88e6307ea2cab372cb675d3e621106e69442856f5e80c227e6afd4de47bcc220"} Oct 30 07:45:52.065738 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:52.065702 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:45:52.065953 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:52.065929 4951 patch_prober.go:29] interesting pod/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 30 07:45:52.066012 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:52.065977 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" podUID="ac54b4ca-6422-4b37-b14c-7a0b8e3657cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 30 07:45:53.066099 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:53.066057 4951 patch_prober.go:29] interesting pod/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 30 07:45:53.066615 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:53.066125 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" podUID="ac54b4ca-6422-4b37-b14c-7a0b8e3657cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 30 07:45:53.067487 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:45:53.067448 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:45:54.067126 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:54.067085 4951 patch_prober.go:29] interesting pod/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 30 07:45:54.067677 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:54.067173 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" podUID="ac54b4ca-6422-4b37-b14c-7a0b8e3657cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 30 07:45:55.065675 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:55.065631 4951 patch_prober.go:29] interesting pod/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 30 07:45:55.065869 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:55.065694 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" podUID="ac54b4ca-6422-4b37-b14c-7a0b8e3657cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 30 07:45:56.065782 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:56.065740 4951 patch_prober.go:29] interesting pod/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 30 07:45:56.066286 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:56.065809 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" podUID="ac54b4ca-6422-4b37-b14c-7a0b8e3657cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 30 07:45:57.137212 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:57.137180 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:45:57.138494 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:57.138464 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:45:57.139522 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:45:57.139503 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:46:07.175265 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:07.175225 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn"] Oct 30 07:46:07.176113 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:07.176092 4951 cpu_manager.go:414] "containerMap: removing container" podUID="a859e7f4-ca02-45e7-b64c-9b71b4d88311" containerName="cleanup" Oct 30 07:46:07.176113 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:07.176111 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a859e7f4-ca02-45e7-b64c-9b71b4d88311" containerName="cleanup" Oct 30 07:46:07.176290 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:07.176132 4951 cpu_manager.go:414] "containerMap: removing container" podUID="bb22b95a-6a2c-4727-80a1-258347acd504" containerName="cleanup" Oct 30 07:46:07.176290 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:07.176141 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb22b95a-6a2c-4727-80a1-258347acd504" containerName="cleanup" Oct 30 07:46:07.176290 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:07.176173 4951 cpu_manager.go:414] "containerMap: removing container" podUID="bb22b95a-6a2c-4727-80a1-258347acd504" containerName="cleanup" Oct 30 07:46:07.176290 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:07.176178 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb22b95a-6a2c-4727-80a1-258347acd504" containerName="cleanup" Oct 30 07:46:07.176290 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:07.176190 4951 cpu_manager.go:414] "containerMap: removing container" podUID="a859e7f4-ca02-45e7-b64c-9b71b4d88311" containerName="cleanup" Oct 30 07:46:07.176290 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:07.176196 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a859e7f4-ca02-45e7-b64c-9b71b4d88311" containerName="cleanup" Oct 30 07:46:07.176290 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:07.176218 4951 cpu_manager.go:414] "containerMap: removing container" podUID="bb22b95a-6a2c-4727-80a1-258347acd504" containerName="cleanup" Oct 30 07:46:07.176290 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:07.176224 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb22b95a-6a2c-4727-80a1-258347acd504" containerName="cleanup" Oct 30 07:46:07.176691 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:07.176345 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb22b95a-6a2c-4727-80a1-258347acd504" containerName="cleanup" Oct 30 07:46:07.176691 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:07.176358 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb22b95a-6a2c-4727-80a1-258347acd504" containerName="cleanup" Oct 30 07:46:07.176691 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:07.176366 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="a859e7f4-ca02-45e7-b64c-9b71b4d88311" containerName="cleanup" Oct 30 07:46:07.176691 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:07.176380 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="a859e7f4-ca02-45e7-b64c-9b71b4d88311" containerName="cleanup" Oct 30 07:46:07.176691 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:07.176391 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="a859e7f4-ca02-45e7-b64c-9b71b4d88311" containerName="cleanup" Oct 30 07:46:07.179750 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:07.179728 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn" Oct 30 07:46:07.212706 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:07.212656 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-fc3a23-serving-cert\"" Oct 30 07:46:07.217458 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:07.217429 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn"] Oct 30 07:46:07.278859 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:07.278756 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvsxp\" (UniqueName: \"kubernetes.io/projected/da8a133c-4e96-4c80-aafa-a08a1082545b-kube-api-access-hvsxp\") pod \"maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn\" (UID: \"da8a133c-4e96-4c80-aafa-a08a1082545b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn" Oct 30 07:46:07.279265 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:07.279236 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da8a133c-4e96-4c80-aafa-a08a1082545b-maas-api-tls\") pod \"maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn\" (UID: \"da8a133c-4e96-4c80-aafa-a08a1082545b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn" Oct 30 07:46:07.279627 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:07.279604 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da8a133c-4e96-4c80-aafa-a08a1082545b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn\" (UID: \"da8a133c-4e96-4c80-aafa-a08a1082545b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn" Oct 30 07:46:07.279800 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:07.279781 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da8a133c-4e96-4c80-aafa-a08a1082545b-metrics-certs\") pod \"maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn\" (UID: \"da8a133c-4e96-4c80-aafa-a08a1082545b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn" Oct 30 07:46:07.415034 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:07.388364 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da8a133c-4e96-4c80-aafa-a08a1082545b-maas-api-tls\") pod \"maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn\" (UID: \"da8a133c-4e96-4c80-aafa-a08a1082545b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn" Oct 30 07:46:07.434781 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:07.434698 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da8a133c-4e96-4c80-aafa-a08a1082545b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn\" (UID: \"da8a133c-4e96-4c80-aafa-a08a1082545b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn" Oct 30 07:46:07.452761 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:07.452722 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da8a133c-4e96-4c80-aafa-a08a1082545b-metrics-certs\") pod \"maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn\" (UID: \"da8a133c-4e96-4c80-aafa-a08a1082545b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn" Oct 30 07:46:07.453905 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:07.453873 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvsxp\" (UniqueName: \"kubernetes.io/projected/da8a133c-4e96-4c80-aafa-a08a1082545b-kube-api-access-hvsxp\") pod \"maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn\" (UID: \"da8a133c-4e96-4c80-aafa-a08a1082545b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn" Oct 30 07:46:07.497802 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:07.431916 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da8a133c-4e96-4c80-aafa-a08a1082545b-maas-api-tls\") pod \"maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn\" (UID: \"da8a133c-4e96-4c80-aafa-a08a1082545b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn" Oct 30 07:46:07.498022 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:07.435653 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da8a133c-4e96-4c80-aafa-a08a1082545b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn\" (UID: \"da8a133c-4e96-4c80-aafa-a08a1082545b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn" Oct 30 07:46:07.498125 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:07.471025 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da8a133c-4e96-4c80-aafa-a08a1082545b-metrics-certs\") pod \"maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn\" (UID: \"da8a133c-4e96-4c80-aafa-a08a1082545b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn" Oct 30 07:46:07.500431 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:07.500402 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvsxp\" (UniqueName: \"kubernetes.io/projected/da8a133c-4e96-4c80-aafa-a08a1082545b-kube-api-access-hvsxp\") pod \"maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn\" (UID: \"da8a133c-4e96-4c80-aafa-a08a1082545b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn" Oct 30 07:46:07.794626 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:07.794583 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn" Oct 30 07:46:08.142762 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:08.142729 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn"] Oct 30 07:46:08.145712 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:46:08.145680 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda8a133c_4e96_4c80_aafa_a08a1082545b.slice/crio-87041a89cc6da35620999557122f02df64f853cda1955d9bc14a5943bff88049 WatchSource:0}: Error finding container 87041a89cc6da35620999557122f02df64f853cda1955d9bc14a5943bff88049: Status 404 returned error can't find the container with id 87041a89cc6da35620999557122f02df64f853cda1955d9bc14a5943bff88049 Oct 30 07:46:08.147797 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:08.147774 4951 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 07:46:08.316543 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:08.316507 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn" event={"ID":"da8a133c-4e96-4c80-aafa-a08a1082545b","Type":"ContainerStarted","Data":"87041a89cc6da35620999557122f02df64f853cda1955d9bc14a5943bff88049"} Oct 30 07:46:09.328496 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:09.328461 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn" event={"ID":"da8a133c-4e96-4c80-aafa-a08a1082545b","Type":"ContainerStarted","Data":"8be600bc8e81c2bbadb72cec0556e43218493feeee46b688099ab831a0fe43fa"} Oct 30 07:46:09.329004 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:09.328555 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn" Oct 30 07:46:09.383391 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:09.383335 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn" podStartSLOduration=1.970925247 podStartE2EDuration="2.383319652s" totalImagesPullingTime="412.394405ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:46:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:46:09.380448435 +0000 UTC m=+2975.723496397" watchObservedRunningTime="2026-10-30 07:46:09.383319652 +0000 UTC m=+2975.726367616" Oct 30 07:46:13.380012 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:13.379978 4951 generic.go:356] "Generic (PLEG): container finished" podID="7771d5de-79de-4e6d-a0e8-acbacaf902af" containerID="e6a584ae28bee7ee12fa124066766c0857e3af029a91b49eec1f1739d75fba0f" exitCode=0 Oct 30 07:46:13.380556 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:13.380051 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" event={"ID":"7771d5de-79de-4e6d-a0e8-acbacaf902af","Type":"ContainerDied","Data":"e6a584ae28bee7ee12fa124066766c0857e3af029a91b49eec1f1739d75fba0f"} Oct 30 07:46:13.380702 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:13.380664 4951 scope.go:122] "RemoveContainer" containerID="e6a584ae28bee7ee12fa124066766c0857e3af029a91b49eec1f1739d75fba0f" Oct 30 07:46:13.382751 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:13.382717 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:46:13.382870 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:13.382806 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:46:13.382870 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:13.382854 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:46:14.389870 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:14.389835 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" event={"ID":"7771d5de-79de-4e6d-a0e8-acbacaf902af","Type":"ContainerStarted","Data":"65968c8fd7ec34701c0910e168db0218fa96979f80b32225dac154ca5e6ebadf"} Oct 30 07:46:14.488655 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:14.488618 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:46:14.489737 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:14.489710 4951 patch_prober.go:29] interesting pod/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Oct 30 07:46:14.489868 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:14.489775 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" podUID="7771d5de-79de-4e6d-a0e8-acbacaf902af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Oct 30 07:46:15.356363 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:15.356330 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn" Oct 30 07:46:15.488822 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:15.488781 4951 patch_prober.go:29] interesting pod/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Oct 30 07:46:15.489409 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:15.488859 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" podUID="7771d5de-79de-4e6d-a0e8-acbacaf902af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Oct 30 07:46:16.489694 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:16.489660 4951 patch_prober.go:29] interesting pod/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Oct 30 07:46:16.490112 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:16.489729 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" podUID="7771d5de-79de-4e6d-a0e8-acbacaf902af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Oct 30 07:46:17.491598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:17.491123 4951 patch_prober.go:29] interesting pod/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Oct 30 07:46:17.491598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:17.491206 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" podUID="7771d5de-79de-4e6d-a0e8-acbacaf902af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Oct 30 07:46:18.204061 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:18.204024 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:46:18.489417 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:18.489322 4951 patch_prober.go:29] interesting pod/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Oct 30 07:46:18.489417 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:18.489389 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" podUID="7771d5de-79de-4e6d-a0e8-acbacaf902af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Oct 30 07:46:19.489038 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:19.489003 4951 patch_prober.go:29] interesting pod/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Oct 30 07:46:19.489470 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:19.489077 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" podUID="7771d5de-79de-4e6d-a0e8-acbacaf902af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Oct 30 07:46:20.495062 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:20.495032 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:46:20.496065 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:20.496035 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:46:23.025846 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:46:23.025802 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:46:32.224469 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.224423 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx"] Oct 30 07:46:32.227754 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.227726 4951 cpu_manager.go:414] "containerMap: removing container" podUID="a859e7f4-ca02-45e7-b64c-9b71b4d88311" containerName="cleanup" Oct 30 07:46:32.227942 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.227927 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a859e7f4-ca02-45e7-b64c-9b71b4d88311" containerName="cleanup" Oct 30 07:46:32.228348 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.228320 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb22b95a-6a2c-4727-80a1-258347acd504" containerName="cleanup" Oct 30 07:46:32.301905 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.301868 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx"] Oct 30 07:46:32.302216 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.301993 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:46:32.349229 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.349196 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-61f6c2-openshift-default-dockercfg-ndt8q\"" Oct 30 07:46:32.535597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.532165 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-credential-socket\") pod \"e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:46:32.535597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.532286 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e611ef5-131d-44f9-848d-3e9ae765f733-istio-podinfo\") pod \"e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:46:32.535597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.532349 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-workload-socket\") pod \"e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:46:32.535597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.532470 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6scf\" (UniqueName: \"kubernetes.io/projected/0e611ef5-131d-44f9-848d-3e9ae765f733-kube-api-access-n6scf\") pod \"e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:46:32.535597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.532526 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-workload-certs\") pod \"e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:46:32.535597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.532673 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e611ef5-131d-44f9-848d-3e9ae765f733-istiod-ca-cert\") pod \"e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:46:32.535597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.532713 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e611ef5-131d-44f9-848d-3e9ae765f733-istio-token\") pod \"e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:46:32.535597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.532743 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-istio-data\") pod \"e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:46:32.535597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.532769 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-istio-envoy\") pod \"e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:46:32.750625 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.736717 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6scf\" (UniqueName: \"kubernetes.io/projected/0e611ef5-131d-44f9-848d-3e9ae765f733-kube-api-access-n6scf\") pod \"e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:46:32.751166 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.751132 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-workload-certs\") pod \"e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:46:32.770362 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.766469 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e611ef5-131d-44f9-848d-3e9ae765f733-istiod-ca-cert\") pod \"e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:46:32.789646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.770710 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e611ef5-131d-44f9-848d-3e9ae765f733-istio-token\") pod \"e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:46:32.789646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.770757 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-istio-data\") pod \"e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:46:32.789646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.770785 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-istio-envoy\") pod \"e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:46:32.789646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.770902 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-credential-socket\") pod \"e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:46:32.789646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.771029 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e611ef5-131d-44f9-848d-3e9ae765f733-istio-podinfo\") pod \"e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:46:32.789646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.771101 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-workload-socket\") pod \"e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:46:32.789646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.771383 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e611ef5-131d-44f9-848d-3e9ae765f733-istiod-ca-cert\") pod \"e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:46:32.789646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.751792 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-workload-certs\") pod \"e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:46:32.789646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.771537 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-workload-socket\") pod \"e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:46:32.789646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.771819 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-credential-socket\") pod \"e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:46:32.789646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.772428 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-istio-data\") pod \"e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:46:32.791189 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.791157 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-istio-envoy\") pod \"e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:46:32.797790 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.792065 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6scf\" (UniqueName: \"kubernetes.io/projected/0e611ef5-131d-44f9-848d-3e9ae765f733-kube-api-access-n6scf\") pod \"e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:46:32.798645 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.798547 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e611ef5-131d-44f9-848d-3e9ae765f733-istio-podinfo\") pod \"e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:46:32.801766 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.801739 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e611ef5-131d-44f9-848d-3e9ae765f733-istio-token\") pod \"e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:46:32.944272 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:32.944226 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:46:33.320454 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:46:33.320424 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e611ef5_131d_44f9_848d_3e9ae765f733.slice/crio-fe0a38bb5d819f10f162f66b412c7391ed73042d62010288e1786498ed8285dd WatchSource:0}: Error finding container fe0a38bb5d819f10f162f66b412c7391ed73042d62010288e1786498ed8285dd: Status 404 returned error can't find the container with id fe0a38bb5d819f10f162f66b412c7391ed73042d62010288e1786498ed8285dd Oct 30 07:46:33.321593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:33.321548 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx"] Oct 30 07:46:33.323264 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:33.323226 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:46:33.323358 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:33.323295 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:46:33.323358 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:33.323334 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:46:33.881762 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:33.881587 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" event={"ID":"0e611ef5-131d-44f9-848d-3e9ae765f733","Type":"ContainerStarted","Data":"9048f23198076171f275a33798853ea0290b4cb69ea58e343c7180f1c21feea9"} Oct 30 07:46:33.881762 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:33.881677 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" event={"ID":"0e611ef5-131d-44f9-848d-3e9ae765f733","Type":"ContainerStarted","Data":"fe0a38bb5d819f10f162f66b412c7391ed73042d62010288e1786498ed8285dd"} Oct 30 07:46:33.945411 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:33.945375 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:46:33.951296 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:33.951247 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" podStartSLOduration=1.9512333960000001 podStartE2EDuration="1.951233396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:46:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:46:33.945959197 +0000 UTC m=+3000.289007156" watchObservedRunningTime="2026-10-30 07:46:33.951233396 +0000 UTC m=+3000.294281363" Oct 30 07:46:34.565626 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:34.564401 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/3.log" Oct 30 07:46:34.570132 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:34.570102 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 07:46:34.575291 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:34.575271 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/3.log" Oct 30 07:46:34.580375 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:34.580350 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 07:46:34.953167 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:34.953063 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:46:35.896235 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:35.896204 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:46:35.897319 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:35.897300 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:46:40.718672 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:40.718628 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-61f6c2-d6f8dd8fd-tb2zd"] Oct 30 07:46:40.854654 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:40.852848 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-61f6c2-d6f8dd8fd-tb2zd" Oct 30 07:46:40.874919 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:40.874722 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-61f6c2\"" Oct 30 07:46:40.894948 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:40.894834 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-61f6c2-dockercfg-9gbnd\"" Oct 30 07:46:40.934161 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:40.918304 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-61f6c2-d6f8dd8fd-tb2zd"] Oct 30 07:46:40.934161 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:40.920619 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9v4w\" (UniqueName: \"kubernetes.io/projected/2e3d3e6e-232d-48ea-a24e-e25812154cb7-kube-api-access-r9v4w\") pod \"payload-processing-e2e-inf-b-61f6c2-d6f8dd8fd-tb2zd\" (UID: \"2e3d3e6e-232d-48ea-a24e-e25812154cb7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-61f6c2-d6f8dd8fd-tb2zd" Oct 30 07:46:40.934161 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:40.920922 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3d3e6e-232d-48ea-a24e-e25812154cb7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-61f6c2-d6f8dd8fd-tb2zd\" (UID: \"2e3d3e6e-232d-48ea-a24e-e25812154cb7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-61f6c2-d6f8dd8fd-tb2zd" Oct 30 07:46:41.023226 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:41.023197 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3d3e6e-232d-48ea-a24e-e25812154cb7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-61f6c2-d6f8dd8fd-tb2zd\" (UID: \"2e3d3e6e-232d-48ea-a24e-e25812154cb7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-61f6c2-d6f8dd8fd-tb2zd" Oct 30 07:46:41.023474 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:41.023448 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9v4w\" (UniqueName: \"kubernetes.io/projected/2e3d3e6e-232d-48ea-a24e-e25812154cb7-kube-api-access-r9v4w\") pod \"payload-processing-e2e-inf-b-61f6c2-d6f8dd8fd-tb2zd\" (UID: \"2e3d3e6e-232d-48ea-a24e-e25812154cb7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-61f6c2-d6f8dd8fd-tb2zd" Oct 30 07:46:41.024098 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:41.024078 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3d3e6e-232d-48ea-a24e-e25812154cb7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-61f6c2-d6f8dd8fd-tb2zd\" (UID: \"2e3d3e6e-232d-48ea-a24e-e25812154cb7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-61f6c2-d6f8dd8fd-tb2zd" Oct 30 07:46:41.048277 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:41.048245 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9v4w\" (UniqueName: \"kubernetes.io/projected/2e3d3e6e-232d-48ea-a24e-e25812154cb7-kube-api-access-r9v4w\") pod \"payload-processing-e2e-inf-b-61f6c2-d6f8dd8fd-tb2zd\" (UID: \"2e3d3e6e-232d-48ea-a24e-e25812154cb7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-61f6c2-d6f8dd8fd-tb2zd" Oct 30 07:46:41.279103 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:41.279004 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-61f6c2-d6f8dd8fd-tb2zd" Oct 30 07:46:41.685975 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:41.685938 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-61f6c2-d6f8dd8fd-tb2zd"] Oct 30 07:46:41.988732 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:41.988636 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-61f6c2-d6f8dd8fd-tb2zd" event={"ID":"2e3d3e6e-232d-48ea-a24e-e25812154cb7","Type":"ContainerStarted","Data":"b6e6c2ae3e038a6bb99b8e2573b28a109b819f5be045748690b9cb55c59006cb"} Oct 30 07:46:41.988732 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:41.988703 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-61f6c2-d6f8dd8fd-tb2zd" event={"ID":"2e3d3e6e-232d-48ea-a24e-e25812154cb7","Type":"ContainerStarted","Data":"1d59aadf7936e64afd7a935db5c1ee0e4c626c3bb20b7a375fe8f41fb825cd0c"} Oct 30 07:46:41.989172 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:41.988740 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-61f6c2-d6f8dd8fd-tb2zd" Oct 30 07:46:42.036146 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:42.036084 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-61f6c2-d6f8dd8fd-tb2zd" podStartSLOduration=2.036064051 podStartE2EDuration="2.036064051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:46:42.032715029 +0000 UTC m=+3008.375762995" watchObservedRunningTime="2026-10-30 07:46:42.036064051 +0000 UTC m=+3008.379112016" Oct 30 07:46:51.901743 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:51.901701 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z"] Oct 30 07:46:51.907193 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:51.907167 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z" Oct 30 07:46:51.929641 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:51.929613 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z"] Oct 30 07:46:51.937421 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:51.937388 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-63f24b-serving-cert\"" Oct 30 07:46:51.939834 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:51.939745 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be63f88f-66b9-43b2-b91c-b572e8cc846d-maas-api-tls\") pod \"maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z\" (UID: \"be63f88f-66b9-43b2-b91c-b572e8cc846d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z" Oct 30 07:46:51.940009 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:51.939867 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwp2v\" (UniqueName: \"kubernetes.io/projected/be63f88f-66b9-43b2-b91c-b572e8cc846d-kube-api-access-xwp2v\") pod \"maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z\" (UID: \"be63f88f-66b9-43b2-b91c-b572e8cc846d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z" Oct 30 07:46:51.940219 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:51.940195 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be63f88f-66b9-43b2-b91c-b572e8cc846d-metrics-certs\") pod \"maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z\" (UID: \"be63f88f-66b9-43b2-b91c-b572e8cc846d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z" Oct 30 07:46:51.940347 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:51.940329 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be63f88f-66b9-43b2-b91c-b572e8cc846d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z\" (UID: \"be63f88f-66b9-43b2-b91c-b572e8cc846d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z" Oct 30 07:46:52.041519 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:52.041480 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be63f88f-66b9-43b2-b91c-b572e8cc846d-metrics-certs\") pod \"maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z\" (UID: \"be63f88f-66b9-43b2-b91c-b572e8cc846d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z" Oct 30 07:46:52.041737 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:52.041547 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be63f88f-66b9-43b2-b91c-b572e8cc846d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z\" (UID: \"be63f88f-66b9-43b2-b91c-b572e8cc846d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z" Oct 30 07:46:52.041849 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:52.041827 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be63f88f-66b9-43b2-b91c-b572e8cc846d-maas-api-tls\") pod \"maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z\" (UID: \"be63f88f-66b9-43b2-b91c-b572e8cc846d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z" Oct 30 07:46:52.041958 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:52.041869 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwp2v\" (UniqueName: \"kubernetes.io/projected/be63f88f-66b9-43b2-b91c-b572e8cc846d-kube-api-access-xwp2v\") pod \"maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z\" (UID: \"be63f88f-66b9-43b2-b91c-b572e8cc846d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z" Oct 30 07:46:52.042398 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:52.042366 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be63f88f-66b9-43b2-b91c-b572e8cc846d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z\" (UID: \"be63f88f-66b9-43b2-b91c-b572e8cc846d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z" Oct 30 07:46:52.044244 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:52.044214 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be63f88f-66b9-43b2-b91c-b572e8cc846d-metrics-certs\") pod \"maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z\" (UID: \"be63f88f-66b9-43b2-b91c-b572e8cc846d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z" Oct 30 07:46:52.044395 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:52.044374 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be63f88f-66b9-43b2-b91c-b572e8cc846d-maas-api-tls\") pod \"maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z\" (UID: \"be63f88f-66b9-43b2-b91c-b572e8cc846d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z" Oct 30 07:46:52.067466 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:52.067433 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwp2v\" (UniqueName: \"kubernetes.io/projected/be63f88f-66b9-43b2-b91c-b572e8cc846d-kube-api-access-xwp2v\") pod \"maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z\" (UID: \"be63f88f-66b9-43b2-b91c-b572e8cc846d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z" Oct 30 07:46:52.221500 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:52.221418 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z" Oct 30 07:46:52.537788 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:46:52.537761 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe63f88f_66b9_43b2_b91c_b572e8cc846d.slice/crio-1dd8da4d1caa8932e9b79bfaa755a4145a35fc724ff5d7405b1f4f4e07f2ab66 WatchSource:0}: Error finding container 1dd8da4d1caa8932e9b79bfaa755a4145a35fc724ff5d7405b1f4f4e07f2ab66: Status 404 returned error can't find the container with id 1dd8da4d1caa8932e9b79bfaa755a4145a35fc724ff5d7405b1f4f4e07f2ab66 Oct 30 07:46:52.541846 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:52.541816 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z"] Oct 30 07:46:53.000228 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:53.000186 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-61f6c2-d6f8dd8fd-tb2zd" Oct 30 07:46:53.026391 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:46:53.026352 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:46:53.266660 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:53.260339 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z" event={"ID":"be63f88f-66b9-43b2-b91c-b572e8cc846d","Type":"ContainerStarted","Data":"d5a2d6959b8416d58c9eb0edf730dd7ac6e0904613dcceb055641dc9404a2e12"} Oct 30 07:46:53.266660 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:53.260407 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z" event={"ID":"be63f88f-66b9-43b2-b91c-b572e8cc846d","Type":"ContainerStarted","Data":"1dd8da4d1caa8932e9b79bfaa755a4145a35fc724ff5d7405b1f4f4e07f2ab66"} Oct 30 07:46:53.266660 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:53.261628 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z" Oct 30 07:46:53.315548 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:46:53.315490 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z" podStartSLOduration=1.9712133870000002 podStartE2EDuration="2.315470575s" totalImagesPullingTime="344.257188ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:46:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:46:53.312858002 +0000 UTC m=+3019.655905967" watchObservedRunningTime="2026-10-30 07:46:53.315470575 +0000 UTC m=+3019.658518541" Oct 30 07:47:00.293729 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:00.293695 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z" Oct 30 07:47:05.753482 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:05.753269 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm"] Oct 30 07:47:05.758733 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:05.758706 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:47:05.785070 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:05.785036 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm"] Oct 30 07:47:05.791497 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:05.791472 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-759ae6-openshift-default-dockercfg-kbznf\"" Oct 30 07:47:05.832599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:05.832220 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-istio-envoy\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:47:05.832599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:05.832276 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-workload-certs\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:47:05.832599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:05.832338 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd14521-e339-4dcb-a824-9d8171138768-istio-token\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:47:05.832599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:05.832387 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd14521-e339-4dcb-a824-9d8171138768-istio-podinfo\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:47:05.832599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:05.832416 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd14521-e339-4dcb-a824-9d8171138768-istiod-ca-cert\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:47:05.832599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:05.832506 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-workload-socket\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:47:05.832599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:05.832586 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-credential-socket\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:47:05.833163 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:05.832640 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-istio-data\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:47:05.833163 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:05.832670 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtgq6\" (UniqueName: \"kubernetes.io/projected/afd14521-e339-4dcb-a824-9d8171138768-kube-api-access-dtgq6\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:47:05.948606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:05.947833 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd14521-e339-4dcb-a824-9d8171138768-istio-token\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:47:05.948606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:05.947922 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd14521-e339-4dcb-a824-9d8171138768-istio-podinfo\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:47:05.948606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:05.947969 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd14521-e339-4dcb-a824-9d8171138768-istiod-ca-cert\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:47:05.948606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:05.948136 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-workload-socket\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:47:05.948606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:05.948240 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-credential-socket\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:47:05.948606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:05.948288 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-istio-data\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:47:05.948606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:05.948322 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtgq6\" (UniqueName: \"kubernetes.io/projected/afd14521-e339-4dcb-a824-9d8171138768-kube-api-access-dtgq6\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:47:05.948606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:05.948508 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-istio-envoy\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:47:05.948606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:05.948556 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-workload-certs\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:47:05.972596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:05.950192 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-workload-socket\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:47:05.977676 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:05.975228 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-credential-socket\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:47:05.977676 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:05.975953 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd14521-e339-4dcb-a824-9d8171138768-istiod-ca-cert\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:47:05.977676 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:05.976221 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-istio-data\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:47:05.977676 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:05.977014 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-workload-certs\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:47:06.025650 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:06.013685 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-istio-envoy\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:47:06.026241 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:06.026196 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd14521-e339-4dcb-a824-9d8171138768-istio-token\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:47:06.027263 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:06.027240 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd14521-e339-4dcb-a824-9d8171138768-istio-podinfo\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:47:06.061606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:06.058276 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtgq6\" (UniqueName: \"kubernetes.io/projected/afd14521-e339-4dcb-a824-9d8171138768-kube-api-access-dtgq6\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:47:06.090536 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:06.090246 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:47:06.656782 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:06.656753 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm"] Oct 30 07:47:06.657308 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:47:06.657280 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd14521_e339_4dcb_a824_9d8171138768.slice/crio-ce8df9d8da833f98c2aaaa3fc9bd28b4407418a873c7d1eb716a56405d8fe2ae WatchSource:0}: Error finding container ce8df9d8da833f98c2aaaa3fc9bd28b4407418a873c7d1eb716a56405d8fe2ae: Status 404 returned error can't find the container with id ce8df9d8da833f98c2aaaa3fc9bd28b4407418a873c7d1eb716a56405d8fe2ae Oct 30 07:47:06.661152 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:06.661116 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:47:06.661241 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:06.661190 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:47:06.661241 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:06.661236 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:47:07.470450 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:07.470400 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" event={"ID":"afd14521-e339-4dcb-a824-9d8171138768","Type":"ContainerStarted","Data":"c3b59c948fa7d3504e0440220a34f6d0171088fae364469414346869baf2f072"} Oct 30 07:47:07.471058 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:07.470466 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" event={"ID":"afd14521-e339-4dcb-a824-9d8171138768","Type":"ContainerStarted","Data":"ce8df9d8da833f98c2aaaa3fc9bd28b4407418a873c7d1eb716a56405d8fe2ae"} Oct 30 07:47:07.532206 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:07.532148 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" podStartSLOduration=2.532132176 podStartE2EDuration="2.532132176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:47:07.531704416 +0000 UTC m=+3033.874752409" watchObservedRunningTime="2026-10-30 07:47:07.532132176 +0000 UTC m=+3033.875180140" Oct 30 07:47:08.091853 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:08.091810 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:47:08.098694 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:08.098662 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:47:08.479706 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:08.479622 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:47:08.480822 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:08.480797 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:47:23.026519 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:47:23.026477 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:47:28.609016 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:28.608983 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx"] Oct 30 07:47:28.613969 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:28.613941 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx" Oct 30 07:47:28.632005 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:28.630968 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-61f6c2-serving-cert\"" Oct 30 07:47:28.678032 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:28.677998 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx"] Oct 30 07:47:28.786720 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:28.786675 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fqnj\" (UniqueName: \"kubernetes.io/projected/e64255a8-c8c4-4ed3-b280-4d35314e746d-kube-api-access-8fqnj\") pod \"maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx\" (UID: \"e64255a8-c8c4-4ed3-b280-4d35314e746d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx" Oct 30 07:47:28.786931 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:28.786757 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e64255a8-c8c4-4ed3-b280-4d35314e746d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx\" (UID: \"e64255a8-c8c4-4ed3-b280-4d35314e746d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx" Oct 30 07:47:28.787665 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:28.787636 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e64255a8-c8c4-4ed3-b280-4d35314e746d-metrics-certs\") pod \"maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx\" (UID: \"e64255a8-c8c4-4ed3-b280-4d35314e746d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx" Oct 30 07:47:28.787900 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:28.787878 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e64255a8-c8c4-4ed3-b280-4d35314e746d-maas-api-tls\") pod \"maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx\" (UID: \"e64255a8-c8c4-4ed3-b280-4d35314e746d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx" Oct 30 07:47:28.914661 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:28.891019 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e64255a8-c8c4-4ed3-b280-4d35314e746d-metrics-certs\") pod \"maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx\" (UID: \"e64255a8-c8c4-4ed3-b280-4d35314e746d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx" Oct 30 07:47:28.915214 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:28.915173 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e64255a8-c8c4-4ed3-b280-4d35314e746d-maas-api-tls\") pod \"maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx\" (UID: \"e64255a8-c8c4-4ed3-b280-4d35314e746d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx" Oct 30 07:47:28.916047 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:28.916021 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fqnj\" (UniqueName: \"kubernetes.io/projected/e64255a8-c8c4-4ed3-b280-4d35314e746d-kube-api-access-8fqnj\") pod \"maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx\" (UID: \"e64255a8-c8c4-4ed3-b280-4d35314e746d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx" Oct 30 07:47:28.916786 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:28.916760 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e64255a8-c8c4-4ed3-b280-4d35314e746d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx\" (UID: \"e64255a8-c8c4-4ed3-b280-4d35314e746d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx" Oct 30 07:47:28.919085 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:28.919059 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e64255a8-c8c4-4ed3-b280-4d35314e746d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx\" (UID: \"e64255a8-c8c4-4ed3-b280-4d35314e746d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx" Oct 30 07:47:28.936606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:28.924185 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e64255a8-c8c4-4ed3-b280-4d35314e746d-maas-api-tls\") pod \"maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx\" (UID: \"e64255a8-c8c4-4ed3-b280-4d35314e746d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx" Oct 30 07:47:28.958594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:28.958544 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e64255a8-c8c4-4ed3-b280-4d35314e746d-metrics-certs\") pod \"maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx\" (UID: \"e64255a8-c8c4-4ed3-b280-4d35314e746d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx" Oct 30 07:47:28.966615 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:28.965309 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fqnj\" (UniqueName: \"kubernetes.io/projected/e64255a8-c8c4-4ed3-b280-4d35314e746d-kube-api-access-8fqnj\") pod \"maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx\" (UID: \"e64255a8-c8c4-4ed3-b280-4d35314e746d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx" Oct 30 07:47:29.231777 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:29.231683 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx" Oct 30 07:47:29.569933 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:47:29.569898 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode64255a8_c8c4_4ed3_b280_4d35314e746d.slice/crio-d5b19ebb3a449e92d2c106463ec22bbe1a07eaca5a99520da24f14550d79da66 WatchSource:0}: Error finding container d5b19ebb3a449e92d2c106463ec22bbe1a07eaca5a99520da24f14550d79da66: Status 404 returned error can't find the container with id d5b19ebb3a449e92d2c106463ec22bbe1a07eaca5a99520da24f14550d79da66 Oct 30 07:47:29.570414 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:29.570389 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx"] Oct 30 07:47:29.704530 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:29.704492 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx" event={"ID":"e64255a8-c8c4-4ed3-b280-4d35314e746d","Type":"ContainerStarted","Data":"d5b19ebb3a449e92d2c106463ec22bbe1a07eaca5a99520da24f14550d79da66"} Oct 30 07:47:30.736784 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:30.736689 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx" event={"ID":"e64255a8-c8c4-4ed3-b280-4d35314e746d","Type":"ContainerStarted","Data":"5978f86e4ff9696fba8e2640b115a35583b785cf45b52f6fa2b4f07bfcc2e1a4"} Oct 30 07:47:30.739044 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:30.738434 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx" Oct 30 07:47:30.789587 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:30.789523 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx" podStartSLOduration=2.347227206 podStartE2EDuration="2.789503351s" totalImagesPullingTime="442.276145ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:47:30.784281816 +0000 UTC m=+3057.127329804" watchObservedRunningTime="2026-10-30 07:47:30.789503351 +0000 UTC m=+3057.132551316" Oct 30 07:47:37.760363 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:47:37.760332 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx" Oct 30 07:47:53.059597 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:47:53.058988 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:48:02.058006 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:02.057972 4951 generic.go:356] "Generic (PLEG): container finished" podID="0e611ef5-131d-44f9-848d-3e9ae765f733" containerID="9048f23198076171f275a33798853ea0290b4cb69ea58e343c7180f1c21feea9" exitCode=0 Oct 30 07:48:02.058506 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:02.058055 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" event={"ID":"0e611ef5-131d-44f9-848d-3e9ae765f733","Type":"ContainerDied","Data":"9048f23198076171f275a33798853ea0290b4cb69ea58e343c7180f1c21feea9"} Oct 30 07:48:02.058623 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:02.058550 4951 scope.go:122] "RemoveContainer" containerID="9048f23198076171f275a33798853ea0290b4cb69ea58e343c7180f1c21feea9" Oct 30 07:48:02.061055 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:02.061005 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:48:02.061179 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:02.061095 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:48:02.061179 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:02.061133 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:48:03.071409 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:03.071369 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" event={"ID":"0e611ef5-131d-44f9-848d-3e9ae765f733","Type":"ContainerStarted","Data":"7c1ddacc8d8bd717f0ef4408be5d4c1f66f326dd427479a5894f5a4292b56ccd"} Oct 30 07:48:03.595731 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:03.595695 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d"] Oct 30 07:48:03.599797 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:03.599772 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d" Oct 30 07:48:03.618442 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:03.618411 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-759ae6-serving-cert\"" Oct 30 07:48:03.678786 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:03.678757 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d"] Oct 30 07:48:03.835621 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:03.796188 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hvc8\" (UniqueName: \"kubernetes.io/projected/e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4-kube-api-access-2hvc8\") pod \"maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d\" (UID: \"e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d" Oct 30 07:48:03.837669 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:03.837635 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4-metrics-certs\") pod \"maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d\" (UID: \"e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d" Oct 30 07:48:03.838004 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:03.837986 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d\" (UID: \"e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d" Oct 30 07:48:03.838218 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:03.838201 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4-maas-api-tls\") pod \"maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d\" (UID: \"e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d" Oct 30 07:48:03.950894 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:03.950786 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4-metrics-certs\") pod \"maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d\" (UID: \"e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d" Oct 30 07:48:03.951218 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:03.951194 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d\" (UID: \"e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d" Oct 30 07:48:03.951379 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:03.951364 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4-maas-api-tls\") pod \"maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d\" (UID: \"e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d" Oct 30 07:48:03.951672 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:03.951655 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hvc8\" (UniqueName: \"kubernetes.io/projected/e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4-kube-api-access-2hvc8\") pod \"maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d\" (UID: \"e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d" Oct 30 07:48:03.953028 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:03.953008 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:48:03.953336 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:48:03.953259 4951 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-759ae6-serving-cert: secret "maas-api-e2e-inf-b-759ae6-serving-cert" not found Oct 30 07:48:03.953511 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:48:03.953500 4951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4-maas-api-tls podName:e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4 nodeName:}" failed. No retries permitted until 2026-10-30 07:48:04.453478234 +0000 UTC m=+3090.796526194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4-maas-api-tls") pod "maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d" (UID: "e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4") : secret "maas-api-e2e-inf-b-759ae6-serving-cert" not found Oct 30 07:48:03.953646 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:03.953529 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d\" (UID: \"e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d" Oct 30 07:48:03.955671 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:03.955563 4951 patch_prober.go:29] interesting pod/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 30 07:48:03.955671 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:03.955636 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" podUID="0e611ef5-131d-44f9-848d-3e9ae765f733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 30 07:48:03.956419 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:03.956389 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4-metrics-certs\") pod \"maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d\" (UID: \"e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d" Oct 30 07:48:03.977188 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:03.977142 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hvc8\" (UniqueName: \"kubernetes.io/projected/e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4-kube-api-access-2hvc8\") pod \"maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d\" (UID: \"e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d" Oct 30 07:48:04.466592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:04.463399 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4-maas-api-tls\") pod \"maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d\" (UID: \"e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d" Oct 30 07:48:04.503591 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:04.503366 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4-maas-api-tls\") pod \"maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d\" (UID: \"e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d" Oct 30 07:48:04.513643 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:04.513615 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d" Oct 30 07:48:05.047948 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:05.017492 4951 patch_prober.go:29] interesting pod/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 30 07:48:05.047948 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:05.018923 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" podUID="0e611ef5-131d-44f9-848d-3e9ae765f733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 30 07:48:05.087273 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:05.086874 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d"] Oct 30 07:48:05.103203 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:05.103142 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d" event={"ID":"e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4","Type":"ContainerStarted","Data":"52c6f5a1c742275bf4efc4486b46afbd97a1ab0b4633cdec5243f09dc18cccc0"} Oct 30 07:48:05.896934 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:05.896902 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:48:05.947597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:05.945887 4951 patch_prober.go:29] interesting pod/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 30 07:48:05.947597 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:05.945957 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" podUID="0e611ef5-131d-44f9-848d-3e9ae765f733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 30 07:48:06.115048 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:06.115015 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d" event={"ID":"e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4","Type":"ContainerStarted","Data":"b9be4a40b792df9fa9e0dbc8fae95f6b1f0d95e6ca07aaad085a730a99ee7190"} Oct 30 07:48:06.115226 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:06.115099 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d" Oct 30 07:48:06.170457 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:06.170336 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d" podStartSLOduration=2.823457813 podStartE2EDuration="3.170320523s" totalImagesPullingTime="346.86271ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:48:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:48:06.16680954 +0000 UTC m=+3092.509857507" watchObservedRunningTime="2026-10-30 07:48:06.170320523 +0000 UTC m=+3092.513368517" Oct 30 07:48:06.952598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:06.947190 4951 patch_prober.go:29] interesting pod/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 30 07:48:06.952598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:06.947251 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" podUID="0e611ef5-131d-44f9-848d-3e9ae765f733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 30 07:48:07.953618 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:07.947991 4951 patch_prober.go:29] interesting pod/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 30 07:48:07.953618 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:07.948054 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" podUID="0e611ef5-131d-44f9-848d-3e9ae765f733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 30 07:48:08.950463 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:08.950425 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:48:08.951604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:08.951560 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:48:12.163054 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:12.163018 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d" Oct 30 07:48:23.033434 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:48:23.033306 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:48:51.123919 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.123879 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m"] Oct 30 07:48:51.129221 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.129195 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:48:51.156412 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.156372 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m"] Oct 30 07:48:51.268545 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.268503 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-credential-socket\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:48:51.268817 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.268613 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istio-data\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:48:51.268817 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.268695 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-workload-certs\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:48:51.268817 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.268762 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istiod-ca-cert\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:48:51.269001 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.268859 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istio-podinfo\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:48:51.269082 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.269062 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istio-token\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:48:51.269181 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.269164 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istio-envoy\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:48:51.269370 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.269344 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nw5r\" (UniqueName: \"kubernetes.io/projected/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-kube-api-access-5nw5r\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:48:51.270195 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.270167 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-workload-socket\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:48:51.392847 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.390913 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istio-envoy\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:48:51.392847 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.391057 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nw5r\" (UniqueName: \"kubernetes.io/projected/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-kube-api-access-5nw5r\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:48:51.392847 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.391118 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-workload-socket\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:48:51.392847 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.391192 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-credential-socket\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:48:51.392847 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.391232 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istio-data\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:48:51.392847 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.391271 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-workload-certs\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:48:51.392847 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.391332 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istiod-ca-cert\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:48:51.392847 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.391552 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istio-podinfo\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:48:51.392847 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.391893 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istio-token\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:48:51.394770 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.393944 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istio-data\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:48:51.395706 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.395017 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-workload-socket\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:48:51.395706 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.395283 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-credential-socket\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:48:51.396253 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.396201 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istiod-ca-cert\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:48:51.396790 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.396772 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-workload-certs\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:48:51.449003 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.448933 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istio-envoy\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:48:51.449794 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.449386 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istio-token\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:48:51.449794 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.449743 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istio-podinfo\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:48:51.449996 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.449979 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nw5r\" (UniqueName: \"kubernetes.io/projected/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-kube-api-access-5nw5r\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:48:51.459146 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.458902 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:48:51.800021 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:48:51.799991 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d7ac8a_0bb5_44a8_8157_aa5ba2f8ec37.slice/crio-7f045dbe2ad7ae31ea0d329682c775233b9bd7cf6948aab571c3d8c1043e1b03 WatchSource:0}: Error finding container 7f045dbe2ad7ae31ea0d329682c775233b9bd7cf6948aab571c3d8c1043e1b03: Status 404 returned error can't find the container with id 7f045dbe2ad7ae31ea0d329682c775233b9bd7cf6948aab571c3d8c1043e1b03 Oct 30 07:48:51.801993 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.801964 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m"] Oct 30 07:48:51.803765 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.803729 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:48:51.803870 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.803794 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:48:51.803870 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:51.803826 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:48:52.011078 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:52.011039 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" event={"ID":"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37","Type":"ContainerStarted","Data":"c69d8059ca8406655ab15ea4adf03915fe2fc118aff65cd6c7ee76b89643a3d1"} Oct 30 07:48:52.011255 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:52.011098 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" event={"ID":"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37","Type":"ContainerStarted","Data":"7f045dbe2ad7ae31ea0d329682c775233b9bd7cf6948aab571c3d8c1043e1b03"} Oct 30 07:48:52.073988 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:52.073877 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" podStartSLOduration=1.0738564 podStartE2EDuration="1.0738564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:48:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:48:52.071654909 +0000 UTC m=+3138.414702877" watchObservedRunningTime="2026-10-30 07:48:52.0738564 +0000 UTC m=+3138.416904367" Oct 30 07:48:52.460260 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:52.460145 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:48:52.462876 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:52.462839 4951 patch_prober.go:29] interesting pod/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 30 07:48:52.463049 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:52.462918 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" podUID="16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 30 07:48:53.036752 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:53.033609 4951 generic.go:356] "Generic (PLEG): container finished" podID="afd14521-e339-4dcb-a824-9d8171138768" containerID="c3b59c948fa7d3504e0440220a34f6d0171088fae364469414346869baf2f072" exitCode=0 Oct 30 07:48:53.036752 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:53.034893 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" event={"ID":"afd14521-e339-4dcb-a824-9d8171138768","Type":"ContainerDied","Data":"c3b59c948fa7d3504e0440220a34f6d0171088fae364469414346869baf2f072"} Oct 30 07:48:53.036752 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:53.035378 4951 scope.go:122] "RemoveContainer" containerID="c3b59c948fa7d3504e0440220a34f6d0171088fae364469414346869baf2f072" Oct 30 07:48:53.046287 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:53.039092 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:48:53.046287 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:53.039191 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:48:53.046287 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:53.039243 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:48:53.062112 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:48:53.062077 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:48:53.462074 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:53.462035 4951 patch_prober.go:29] interesting pod/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 30 07:48:53.462647 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:53.462101 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" podUID="16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 30 07:48:53.480409 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:53.480358 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:48:54.067774 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:54.067730 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" event={"ID":"afd14521-e339-4dcb-a824-9d8171138768","Type":"ContainerStarted","Data":"7501eddda9930e9bf0465095e72a76ad00cc2aadb5553635f5000d54784ba1ed"} Oct 30 07:48:54.091936 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:54.091884 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:48:54.093270 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:54.093249 4951 patch_prober.go:29] interesting pod/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 30 07:48:54.093378 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:54.093302 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" podUID="afd14521-e339-4dcb-a824-9d8171138768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 30 07:48:54.460198 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:54.460117 4951 patch_prober.go:29] interesting pod/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 30 07:48:54.460416 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:54.460393 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" podUID="16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 30 07:48:55.093444 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:55.093407 4951 patch_prober.go:29] interesting pod/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 30 07:48:55.094076 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:55.094041 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" podUID="afd14521-e339-4dcb-a824-9d8171138768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 30 07:48:55.481710 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:55.460992 4951 patch_prober.go:29] interesting pod/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 30 07:48:55.481710 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:55.461083 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" podUID="16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 30 07:48:56.091386 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:56.091349 4951 patch_prober.go:29] interesting pod/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 30 07:48:56.091567 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:56.091397 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" podUID="afd14521-e339-4dcb-a824-9d8171138768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 30 07:48:56.472091 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:56.471999 4951 patch_prober.go:29] interesting pod/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 30 07:48:56.475314 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:56.475272 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" podUID="16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 30 07:48:57.091177 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:57.091130 4951 patch_prober.go:29] interesting pod/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 30 07:48:57.091359 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:57.091194 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" podUID="afd14521-e339-4dcb-a824-9d8171138768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 30 07:48:57.508661 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:57.508623 4951 patch_prober.go:29] interesting pod/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 30 07:48:57.509283 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:57.509250 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" podUID="16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 30 07:48:58.092168 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:58.092128 4951 patch_prober.go:29] interesting pod/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 30 07:48:58.092360 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:58.092183 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" podUID="afd14521-e339-4dcb-a824-9d8171138768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 30 07:48:58.465927 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:58.465848 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:48:58.466688 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:58.466668 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:48:58.467638 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:58.467618 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:48:59.092083 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:59.092047 4951 patch_prober.go:29] interesting pod/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 30 07:48:59.092492 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:48:59.092116 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" podUID="afd14521-e339-4dcb-a824-9d8171138768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 30 07:49:00.092107 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:49:00.092069 4951 patch_prober.go:29] interesting pod/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 30 07:49:00.092502 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:49:00.092138 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" podUID="afd14521-e339-4dcb-a824-9d8171138768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 30 07:49:01.097968 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:49:01.097934 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:49:01.098434 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:49:01.098417 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:49:01.099347 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:49:01.099327 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:49:23.026786 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:49:23.026748 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:49:40.509618 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:49:40.507667 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:49:40.615825 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:49:40.614666 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:49:40.633517 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:49:40.632713 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:49:41.207622 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:49:41.204674 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:49:41.208298 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:49:41.208257 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:49:53.027025 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:49:53.026981 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:50:23.028018 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:50:23.027981 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:50:53.027543 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:50:53.027504 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:51:22.624410 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:22.624325 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-352fb3-67f655f77f-xzf5w"] Oct 30 07:51:22.630059 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:22.630033 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-352fb3-67f655f77f-xzf5w" Oct 30 07:51:22.652355 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:22.652323 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-352fb3-67f655f77f-xzf5w"] Oct 30 07:51:22.656849 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:22.656808 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-352fb3-dockercfg-vq7mw\"" Oct 30 07:51:22.657036 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:22.656924 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-352fb3\"" Oct 30 07:51:22.815599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:22.790543 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-352fb3-67f655f77f-xzf5w\" (UID: \"cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-352fb3-67f655f77f-xzf5w" Oct 30 07:51:22.833078 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:22.833028 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jk29\" (UniqueName: \"kubernetes.io/projected/cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2-kube-api-access-5jk29\") pod \"payload-pre-processing-e2e-inf-b-352fb3-67f655f77f-xzf5w\" (UID: \"cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-352fb3-67f655f77f-xzf5w" Oct 30 07:51:23.053664 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:23.053425 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-352fb3-67f655f77f-xzf5w\" (UID: \"cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-352fb3-67f655f77f-xzf5w" Oct 30 07:51:23.056205 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:23.056172 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jk29\" (UniqueName: \"kubernetes.io/projected/cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2-kube-api-access-5jk29\") pod \"payload-pre-processing-e2e-inf-b-352fb3-67f655f77f-xzf5w\" (UID: \"cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-352fb3-67f655f77f-xzf5w" Oct 30 07:51:23.057129 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:23.055630 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-352fb3-67f655f77f-xzf5w\" (UID: \"cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-352fb3-67f655f77f-xzf5w" Oct 30 07:51:23.167385 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:23.167343 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jk29\" (UniqueName: \"kubernetes.io/projected/cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2-kube-api-access-5jk29\") pod \"payload-pre-processing-e2e-inf-b-352fb3-67f655f77f-xzf5w\" (UID: \"cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-352fb3-67f655f77f-xzf5w" Oct 30 07:51:23.197200 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:51:23.197162 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:51:23.224799 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:23.221649 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-352fb3-589f6cb4d8-sdbxs"] Oct 30 07:51:23.229391 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:23.229326 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-352fb3-589f6cb4d8-sdbxs" Oct 30 07:51:23.235464 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:23.235427 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-352fb3-589f6cb4d8-sdbxs"] Oct 30 07:51:23.251235 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:23.251200 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-352fb3-67f655f77f-xzf5w" Oct 30 07:51:23.366510 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:23.366462 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9mcc\" (UniqueName: \"kubernetes.io/projected/1ad0084a-1d54-4b23-ac5c-99cbc8b6da24-kube-api-access-q9mcc\") pod \"payload-processing-e2e-inf-b-352fb3-589f6cb4d8-sdbxs\" (UID: \"1ad0084a-1d54-4b23-ac5c-99cbc8b6da24\") " pod="openshift-ingress/payload-processing-e2e-inf-b-352fb3-589f6cb4d8-sdbxs" Oct 30 07:51:23.366814 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:23.366789 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad0084a-1d54-4b23-ac5c-99cbc8b6da24-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-352fb3-589f6cb4d8-sdbxs\" (UID: \"1ad0084a-1d54-4b23-ac5c-99cbc8b6da24\") " pod="openshift-ingress/payload-processing-e2e-inf-b-352fb3-589f6cb4d8-sdbxs" Oct 30 07:51:23.468315 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:23.468284 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9mcc\" (UniqueName: \"kubernetes.io/projected/1ad0084a-1d54-4b23-ac5c-99cbc8b6da24-kube-api-access-q9mcc\") pod \"payload-processing-e2e-inf-b-352fb3-589f6cb4d8-sdbxs\" (UID: \"1ad0084a-1d54-4b23-ac5c-99cbc8b6da24\") " pod="openshift-ingress/payload-processing-e2e-inf-b-352fb3-589f6cb4d8-sdbxs" Oct 30 07:51:23.468507 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:23.468489 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad0084a-1d54-4b23-ac5c-99cbc8b6da24-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-352fb3-589f6cb4d8-sdbxs\" (UID: \"1ad0084a-1d54-4b23-ac5c-99cbc8b6da24\") " pod="openshift-ingress/payload-processing-e2e-inf-b-352fb3-589f6cb4d8-sdbxs" Oct 30 07:51:23.469207 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:23.469188 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad0084a-1d54-4b23-ac5c-99cbc8b6da24-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-352fb3-589f6cb4d8-sdbxs\" (UID: \"1ad0084a-1d54-4b23-ac5c-99cbc8b6da24\") " pod="openshift-ingress/payload-processing-e2e-inf-b-352fb3-589f6cb4d8-sdbxs" Oct 30 07:51:23.493260 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:23.493234 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9mcc\" (UniqueName: \"kubernetes.io/projected/1ad0084a-1d54-4b23-ac5c-99cbc8b6da24-kube-api-access-q9mcc\") pod \"payload-processing-e2e-inf-b-352fb3-589f6cb4d8-sdbxs\" (UID: \"1ad0084a-1d54-4b23-ac5c-99cbc8b6da24\") " pod="openshift-ingress/payload-processing-e2e-inf-b-352fb3-589f6cb4d8-sdbxs" Oct 30 07:51:23.552030 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:23.551990 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-352fb3-589f6cb4d8-sdbxs" Oct 30 07:51:23.577644 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:23.577598 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-352fb3-67f655f77f-xzf5w"] Oct 30 07:51:23.578666 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:51:23.578636 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea9fef6_4eaf_451c_aee0_8d0d2c70c2a2.slice/crio-65cc4373555bea2d0cca122a1890c5993392d12e0399d7dab1dcb4a4112b2bda WatchSource:0}: Error finding container 65cc4373555bea2d0cca122a1890c5993392d12e0399d7dab1dcb4a4112b2bda: Status 404 returned error can't find the container with id 65cc4373555bea2d0cca122a1890c5993392d12e0399d7dab1dcb4a4112b2bda Oct 30 07:51:23.912194 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:23.910664 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-352fb3-589f6cb4d8-sdbxs"] Oct 30 07:51:24.157391 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:24.157289 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-352fb3-67f655f77f-xzf5w" event={"ID":"cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2","Type":"ContainerStarted","Data":"38d12521fb5c6c835cbc258c388583398a6b5fd114e8ac2ce3ebf1193e5bf67c"} Oct 30 07:51:24.157391 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:24.157344 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-352fb3-67f655f77f-xzf5w" event={"ID":"cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2","Type":"ContainerStarted","Data":"65cc4373555bea2d0cca122a1890c5993392d12e0399d7dab1dcb4a4112b2bda"} Oct 30 07:51:24.157666 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:24.157398 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-352fb3-67f655f77f-xzf5w" Oct 30 07:51:24.159315 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:24.159290 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-352fb3-589f6cb4d8-sdbxs" event={"ID":"1ad0084a-1d54-4b23-ac5c-99cbc8b6da24","Type":"ContainerStarted","Data":"31b368458fa1f8450997a7f6c62de962b68332528d496f8d974c1f0bf50aa995"} Oct 30 07:51:24.159466 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:24.159328 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-352fb3-589f6cb4d8-sdbxs" event={"ID":"1ad0084a-1d54-4b23-ac5c-99cbc8b6da24","Type":"ContainerStarted","Data":"741df68671dba5549c2cfdbae6cd08d48c344eee4cde2c7496f18181db0069d7"} Oct 30 07:51:24.159466 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:24.159365 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-352fb3-589f6cb4d8-sdbxs" Oct 30 07:51:24.207770 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:24.207703 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-352fb3-67f655f77f-xzf5w" podStartSLOduration=2.207683185 podStartE2EDuration="2.207683185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:51:24.202876535 +0000 UTC m=+3290.545924500" watchObservedRunningTime="2026-10-30 07:51:24.207683185 +0000 UTC m=+3290.550731151" Oct 30 07:51:24.246125 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:24.246074 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-352fb3-589f6cb4d8-sdbxs" podStartSLOduration=2.246058154 podStartE2EDuration="2.246058154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:51:24.245353508 +0000 UTC m=+3290.588401475" watchObservedRunningTime="2026-10-30 07:51:24.246058154 +0000 UTC m=+3290.589106119" Oct 30 07:51:34.626030 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:34.625999 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/3.log" Oct 30 07:51:34.632110 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:34.632082 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 07:51:34.644166 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:34.644138 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/3.log" Oct 30 07:51:34.650122 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:34.650097 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 07:51:35.396735 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:35.396699 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-352fb3-589f6cb4d8-sdbxs" Oct 30 07:51:35.396931 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:51:35.396769 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-352fb3-67f655f77f-xzf5w" Oct 30 07:51:53.027939 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:51:53.027892 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:52:03.778101 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:03.778064 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff"] Oct 30 07:52:03.778799 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:03.778347 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" podUID="69b69ee1-c4d2-4502-aea6-d9cd9716034e" containerName="istio-proxy" containerID="cri-o://232990c27293634575c642b4d3af6a78380a4de5fe320a055a75b77f77aeac13" gracePeriod=30 Oct 30 07:52:09.074796 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.074762 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:52:09.145463 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.145428 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istio-podinfo\") pod \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " Oct 30 07:52:09.145960 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.145926 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istiod-ca-cert" pod "69b69ee1-c4d2-4502-aea6-d9cd9716034e" (UID: "69b69ee1-c4d2-4502-aea6-d9cd9716034e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:52:09.146093 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.145992 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istiod-ca-cert\") pod \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " Oct 30 07:52:09.146093 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.146067 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-workload-socket\") pod \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " Oct 30 07:52:09.146386 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.146352 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-workload-socket" pod "69b69ee1-c4d2-4502-aea6-d9cd9716034e" (UID: "69b69ee1-c4d2-4502-aea6-d9cd9716034e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:52:09.146840 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.146809 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istio-data" pod "69b69ee1-c4d2-4502-aea6-d9cd9716034e" (UID: "69b69ee1-c4d2-4502-aea6-d9cd9716034e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:52:09.147016 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.146562 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istio-data\") pod \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " Oct 30 07:52:09.147248 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.147225 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-workload-certs\") pod \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " Oct 30 07:52:09.147516 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.147490 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-workload-certs" pod "69b69ee1-c4d2-4502-aea6-d9cd9716034e" (UID: "69b69ee1-c4d2-4502-aea6-d9cd9716034e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:52:09.147654 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.147548 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-credential-socket\") pod \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " Oct 30 07:52:09.147735 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.147717 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istio-envoy\") pod \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " Oct 30 07:52:09.147785 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.147767 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69b69ee1-c4d2-4502-aea6-d9cd9716034e-kube-api-access-z2s5v\" (UniqueName: \"kubernetes.io/projected/69b69ee1-c4d2-4502-aea6-d9cd9716034e-kube-api-access-z2s5v\") pod \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " Oct 30 07:52:09.147828 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.147815 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-credential-socket" pod "69b69ee1-c4d2-4502-aea6-d9cd9716034e" (UID: "69b69ee1-c4d2-4502-aea6-d9cd9716034e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:52:09.147882 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.147830 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istio-token\" (UniqueName: \"kubernetes.io/projected/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istio-token\") pod \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\" (UID: \"69b69ee1-c4d2-4502-aea6-d9cd9716034e\") " Oct 30 07:52:09.148784 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.148755 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:52:09.148922 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.148909 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:52:09.149083 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.149016 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:52:09.149083 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.149035 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:52:09.149083 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.149059 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:52:09.150336 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.150309 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istio-podinfo" pod "69b69ee1-c4d2-4502-aea6-d9cd9716034e" (UID: "69b69ee1-c4d2-4502-aea6-d9cd9716034e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:52:09.150428 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.150405 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istio-envoy" pod "69b69ee1-c4d2-4502-aea6-d9cd9716034e" (UID: "69b69ee1-c4d2-4502-aea6-d9cd9716034e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:52:09.150472 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.150423 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69b69ee1-c4d2-4502-aea6-d9cd9716034e-kube-api-access-z2s5v" pod "69b69ee1-c4d2-4502-aea6-d9cd9716034e" (UID: "69b69ee1-c4d2-4502-aea6-d9cd9716034e"). InnerVolumeSpecName "kube-api-access-z2s5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:52:09.151052 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.151032 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istio-token" pod "69b69ee1-c4d2-4502-aea6-d9cd9716034e" (UID: "69b69ee1-c4d2-4502-aea6-d9cd9716034e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:52:09.261682 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.261564 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:52:09.261682 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.261616 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2s5v\" (UniqueName: \"kubernetes.io/projected/69b69ee1-c4d2-4502-aea6-d9cd9716034e-kube-api-access-z2s5v\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:52:09.261682 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.261634 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:52:09.261682 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.261647 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69b69ee1-c4d2-4502-aea6-d9cd9716034e-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:52:09.880013 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.879925 4951 generic.go:356] "Generic (PLEG): container finished" podID="69b69ee1-c4d2-4502-aea6-d9cd9716034e" containerID="232990c27293634575c642b4d3af6a78380a4de5fe320a055a75b77f77aeac13" exitCode=0 Oct 30 07:52:09.880013 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.880004 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" event={"ID":"69b69ee1-c4d2-4502-aea6-d9cd9716034e","Type":"ContainerDied","Data":"232990c27293634575c642b4d3af6a78380a4de5fe320a055a75b77f77aeac13"} Oct 30 07:52:09.880229 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.880057 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" event={"ID":"69b69ee1-c4d2-4502-aea6-d9cd9716034e","Type":"ContainerDied","Data":"a838bc5e6454bb4765e39a213063b30bd7e71832ae1a7ec203774cf50b89846d"} Oct 30 07:52:09.880229 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.879980 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff" Oct 30 07:52:09.880229 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.880080 4951 scope.go:122] "RemoveContainer" containerID="232990c27293634575c642b4d3af6a78380a4de5fe320a055a75b77f77aeac13" Oct 30 07:52:09.891905 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.891887 4951 scope.go:122] "RemoveContainer" containerID="232990c27293634575c642b4d3af6a78380a4de5fe320a055a75b77f77aeac13" Oct 30 07:52:09.892176 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:52:09.892158 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"232990c27293634575c642b4d3af6a78380a4de5fe320a055a75b77f77aeac13\": container with ID starting with 232990c27293634575c642b4d3af6a78380a4de5fe320a055a75b77f77aeac13 not found: ID does not exist" containerID="232990c27293634575c642b4d3af6a78380a4de5fe320a055a75b77f77aeac13" Oct 30 07:52:09.892217 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.892188 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"232990c27293634575c642b4d3af6a78380a4de5fe320a055a75b77f77aeac13"} err="failed to get container status \"232990c27293634575c642b4d3af6a78380a4de5fe320a055a75b77f77aeac13\": rpc error: code = NotFound desc = could not find container \"232990c27293634575c642b4d3af6a78380a4de5fe320a055a75b77f77aeac13\": container with ID starting with 232990c27293634575c642b4d3af6a78380a4de5fe320a055a75b77f77aeac13 not found: ID does not exist" Oct 30 07:52:09.956040 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.956001 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff"] Oct 30 07:52:09.968790 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:09.968758 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-s9mff"] Oct 30 07:52:11.775550 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:52:11.775511 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69b69ee1-c4d2-4502-aea6-d9cd9716034e" path="/var/lib/kubelet/pods/69b69ee1-c4d2-4502-aea6-d9cd9716034e/volumes" Oct 30 07:52:23.025533 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:52:23.025498 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:52:53.029916 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:52:53.029854 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:53:23.026141 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:53:23.026099 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:53:47.692141 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:47.692103 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd"] Oct 30 07:53:47.692899 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:47.692343 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" podUID="ac54b4ca-6422-4b37-b14c-7a0b8e3657cc" containerName="istio-proxy" containerID="cri-o://88e6307ea2cab372cb675d3e621106e69442856f5e80c227e6afd4de47bcc220" gracePeriod=30 Oct 30 07:53:53.032055 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.031538 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:53:53.038749 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:53:53.038646 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:53:53.063918 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.063879 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-workload-certs\") pod \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " Oct 30 07:53:53.064225 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.064141 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istio-token\" (UniqueName: \"kubernetes.io/projected/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istio-token\") pod \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " Oct 30 07:53:53.064359 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.064255 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-workload-socket\") pod \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " Oct 30 07:53:53.064438 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.064410 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istio-envoy\") pod \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " Oct 30 07:53:53.066916 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.064997 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-credential-socket\") pod \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " Oct 30 07:53:53.066916 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.065141 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-workload-certs" pod "ac54b4ca-6422-4b37-b14c-7a0b8e3657cc" (UID: "ac54b4ca-6422-4b37-b14c-7a0b8e3657cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:53:53.066916 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.065342 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istiod-ca-cert\") pod \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " Oct 30 07:53:53.066916 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.065400 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-kube-api-access-c9ht5\" (UniqueName: \"kubernetes.io/projected/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-kube-api-access-c9ht5\") pod \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " Oct 30 07:53:53.066916 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.065534 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istio-podinfo\") pod \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " Oct 30 07:53:53.066916 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.065604 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istio-data\") pod \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\" (UID: \"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc\") " Oct 30 07:53:53.066916 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.065651 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istiod-ca-cert" pod "ac54b4ca-6422-4b37-b14c-7a0b8e3657cc" (UID: "ac54b4ca-6422-4b37-b14c-7a0b8e3657cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:53:53.066916 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.066557 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:53:53.066916 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.066599 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:53:53.067771 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.067681 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-credential-socket" pod "ac54b4ca-6422-4b37-b14c-7a0b8e3657cc" (UID: "ac54b4ca-6422-4b37-b14c-7a0b8e3657cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:53:53.067771 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.067751 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istio-data" pod "ac54b4ca-6422-4b37-b14c-7a0b8e3657cc" (UID: "ac54b4ca-6422-4b37-b14c-7a0b8e3657cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:53:53.068592 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.068548 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-workload-socket" pod "ac54b4ca-6422-4b37-b14c-7a0b8e3657cc" (UID: "ac54b4ca-6422-4b37-b14c-7a0b8e3657cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:53:53.091595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.069878 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-kube-api-access-c9ht5" pod "ac54b4ca-6422-4b37-b14c-7a0b8e3657cc" (UID: "ac54b4ca-6422-4b37-b14c-7a0b8e3657cc"). InnerVolumeSpecName "kube-api-access-c9ht5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:53:53.147600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.133713 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istio-envoy" pod "ac54b4ca-6422-4b37-b14c-7a0b8e3657cc" (UID: "ac54b4ca-6422-4b37-b14c-7a0b8e3657cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:53:53.147600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.143601 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istio-token" pod "ac54b4ca-6422-4b37-b14c-7a0b8e3657cc" (UID: "ac54b4ca-6422-4b37-b14c-7a0b8e3657cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:53:53.171383 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.171343 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:53:53.171383 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.171378 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:53:53.171383 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.171388 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9ht5\" (UniqueName: \"kubernetes.io/projected/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-kube-api-access-c9ht5\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:53:53.171721 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.171400 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:53:53.171721 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.171409 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:53:53.171721 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.171421 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:53:53.177985 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.177926 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istio-podinfo" pod "ac54b4ca-6422-4b37-b14c-7a0b8e3657cc" (UID: "ac54b4ca-6422-4b37-b14c-7a0b8e3657cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:53:53.273009 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.272968 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:53:53.334639 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.334560 4951 generic.go:356] "Generic (PLEG): container finished" podID="ac54b4ca-6422-4b37-b14c-7a0b8e3657cc" containerID="88e6307ea2cab372cb675d3e621106e69442856f5e80c227e6afd4de47bcc220" exitCode=0 Oct 30 07:53:53.334845 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.334647 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" Oct 30 07:53:53.334845 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.334602 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" event={"ID":"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc","Type":"ContainerDied","Data":"88e6307ea2cab372cb675d3e621106e69442856f5e80c227e6afd4de47bcc220"} Oct 30 07:53:53.334845 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.334718 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd" event={"ID":"ac54b4ca-6422-4b37-b14c-7a0b8e3657cc","Type":"ContainerDied","Data":"5f26d507e89a728fc1c32bde27c8584c8ae28939acdc178a9d865b4e40a9c536"} Oct 30 07:53:53.334845 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.334748 4951 scope.go:122] "RemoveContainer" containerID="88e6307ea2cab372cb675d3e621106e69442856f5e80c227e6afd4de47bcc220" Oct 30 07:53:53.349195 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.349174 4951 scope.go:122] "RemoveContainer" containerID="139501cb783671fd2c36975c9c1a71091246a217a8862ca1ca657566557ec956" Oct 30 07:53:53.362029 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.361993 4951 scope.go:122] "RemoveContainer" containerID="88e6307ea2cab372cb675d3e621106e69442856f5e80c227e6afd4de47bcc220" Oct 30 07:53:53.362330 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:53:53.362303 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88e6307ea2cab372cb675d3e621106e69442856f5e80c227e6afd4de47bcc220\": container with ID starting with 88e6307ea2cab372cb675d3e621106e69442856f5e80c227e6afd4de47bcc220 not found: ID does not exist" containerID="88e6307ea2cab372cb675d3e621106e69442856f5e80c227e6afd4de47bcc220" Oct 30 07:53:53.362415 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.362339 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88e6307ea2cab372cb675d3e621106e69442856f5e80c227e6afd4de47bcc220"} err="failed to get container status \"88e6307ea2cab372cb675d3e621106e69442856f5e80c227e6afd4de47bcc220\": rpc error: code = NotFound desc = could not find container \"88e6307ea2cab372cb675d3e621106e69442856f5e80c227e6afd4de47bcc220\": container with ID starting with 88e6307ea2cab372cb675d3e621106e69442856f5e80c227e6afd4de47bcc220 not found: ID does not exist" Oct 30 07:53:53.362415 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.362357 4951 scope.go:122] "RemoveContainer" containerID="139501cb783671fd2c36975c9c1a71091246a217a8862ca1ca657566557ec956" Oct 30 07:53:53.362762 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:53:53.362740 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"139501cb783671fd2c36975c9c1a71091246a217a8862ca1ca657566557ec956\": container with ID starting with 139501cb783671fd2c36975c9c1a71091246a217a8862ca1ca657566557ec956 not found: ID does not exist" containerID="139501cb783671fd2c36975c9c1a71091246a217a8862ca1ca657566557ec956" Oct 30 07:53:53.362875 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.362770 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"139501cb783671fd2c36975c9c1a71091246a217a8862ca1ca657566557ec956"} err="failed to get container status \"139501cb783671fd2c36975c9c1a71091246a217a8862ca1ca657566557ec956\": rpc error: code = NotFound desc = could not find container \"139501cb783671fd2c36975c9c1a71091246a217a8862ca1ca657566557ec956\": container with ID starting with 139501cb783671fd2c36975c9c1a71091246a217a8862ca1ca657566557ec956 not found: ID does not exist" Oct 30 07:53:53.393988 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.393949 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd"] Oct 30 07:53:53.398277 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.398249 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-vsdmd"] Oct 30 07:53:53.774370 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:53:53.774341 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac54b4ca-6422-4b37-b14c-7a0b8e3657cc" path="/var/lib/kubelet/pods/ac54b4ca-6422-4b37-b14c-7a0b8e3657cc/volumes" Oct 30 07:54:04.543015 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.542976 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47"] Oct 30 07:54:04.544069 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.544045 4951 cpu_manager.go:414] "containerMap: removing container" podUID="ac54b4ca-6422-4b37-b14c-7a0b8e3657cc" containerName="istio-proxy" Oct 30 07:54:04.544069 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.544068 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac54b4ca-6422-4b37-b14c-7a0b8e3657cc" containerName="istio-proxy" Oct 30 07:54:04.544263 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.544088 4951 cpu_manager.go:414] "containerMap: removing container" podUID="ac54b4ca-6422-4b37-b14c-7a0b8e3657cc" containerName="istio-proxy" Oct 30 07:54:04.544263 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.544094 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac54b4ca-6422-4b37-b14c-7a0b8e3657cc" containerName="istio-proxy" Oct 30 07:54:04.544263 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.544134 4951 cpu_manager.go:414] "containerMap: removing container" podUID="69b69ee1-c4d2-4502-aea6-d9cd9716034e" containerName="istio-proxy" Oct 30 07:54:04.544263 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.544143 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69b69ee1-c4d2-4502-aea6-d9cd9716034e" containerName="istio-proxy" Oct 30 07:54:04.544490 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.544268 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac54b4ca-6422-4b37-b14c-7a0b8e3657cc" containerName="istio-proxy" Oct 30 07:54:04.544490 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.544277 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac54b4ca-6422-4b37-b14c-7a0b8e3657cc" containerName="istio-proxy" Oct 30 07:54:04.544490 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.544296 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="69b69ee1-c4d2-4502-aea6-d9cd9716034e" containerName="istio-proxy" Oct 30 07:54:04.550012 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.549986 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47" Oct 30 07:54:04.571243 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.571214 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47"] Oct 30 07:54:04.577647 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.577619 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-51147f\"/\"kube-root-ca.crt\"" Oct 30 07:54:04.577785 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.577709 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-51147f\"/\"default-dockercfg-rjl8j\"" Oct 30 07:54:04.582561 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.582540 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-51147f\"/\"openshift-service-ca.crt\"" Oct 30 07:54:04.582818 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.582803 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-51147f\"/\"test-model-51147f-kserve-self-signed-certs\"" Oct 30 07:54:04.670229 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.670183 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d50edaa2-d402-4678-8d8b-e265ab2c65ed-tls-certs\") pod \"test-model-51147f-kserve-798f55f75-4gz47\" (UID: \"d50edaa2-d402-4678-8d8b-e265ab2c65ed\") " pod="ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47" Oct 30 07:54:04.670417 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.670309 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d50edaa2-d402-4678-8d8b-e265ab2c65ed-home\") pod \"test-model-51147f-kserve-798f55f75-4gz47\" (UID: \"d50edaa2-d402-4678-8d8b-e265ab2c65ed\") " pod="ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47" Oct 30 07:54:04.670417 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.670380 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d50edaa2-d402-4678-8d8b-e265ab2c65ed-model-cache\") pod \"test-model-51147f-kserve-798f55f75-4gz47\" (UID: \"d50edaa2-d402-4678-8d8b-e265ab2c65ed\") " pod="ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47" Oct 30 07:54:04.670518 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.670492 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d50edaa2-d402-4678-8d8b-e265ab2c65ed-dshm\") pod \"test-model-51147f-kserve-798f55f75-4gz47\" (UID: \"d50edaa2-d402-4678-8d8b-e265ab2c65ed\") " pod="ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47" Oct 30 07:54:04.670604 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.670557 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvsrc\" (UniqueName: \"kubernetes.io/projected/d50edaa2-d402-4678-8d8b-e265ab2c65ed-kube-api-access-rvsrc\") pod \"test-model-51147f-kserve-798f55f75-4gz47\" (UID: \"d50edaa2-d402-4678-8d8b-e265ab2c65ed\") " pod="ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47" Oct 30 07:54:04.670655 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.670631 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d50edaa2-d402-4678-8d8b-e265ab2c65ed-tmp-dir\") pod \"test-model-51147f-kserve-798f55f75-4gz47\" (UID: \"d50edaa2-d402-4678-8d8b-e265ab2c65ed\") " pod="ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47" Oct 30 07:54:04.771914 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.771884 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d50edaa2-d402-4678-8d8b-e265ab2c65ed-home\") pod \"test-model-51147f-kserve-798f55f75-4gz47\" (UID: \"d50edaa2-d402-4678-8d8b-e265ab2c65ed\") " pod="ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47" Oct 30 07:54:04.772088 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.771952 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d50edaa2-d402-4678-8d8b-e265ab2c65ed-model-cache\") pod \"test-model-51147f-kserve-798f55f75-4gz47\" (UID: \"d50edaa2-d402-4678-8d8b-e265ab2c65ed\") " pod="ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47" Oct 30 07:54:04.772088 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.772034 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d50edaa2-d402-4678-8d8b-e265ab2c65ed-dshm\") pod \"test-model-51147f-kserve-798f55f75-4gz47\" (UID: \"d50edaa2-d402-4678-8d8b-e265ab2c65ed\") " pod="ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47" Oct 30 07:54:04.772177 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.772102 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvsrc\" (UniqueName: \"kubernetes.io/projected/d50edaa2-d402-4678-8d8b-e265ab2c65ed-kube-api-access-rvsrc\") pod \"test-model-51147f-kserve-798f55f75-4gz47\" (UID: \"d50edaa2-d402-4678-8d8b-e265ab2c65ed\") " pod="ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47" Oct 30 07:54:04.772177 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.772134 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d50edaa2-d402-4678-8d8b-e265ab2c65ed-tmp-dir\") pod \"test-model-51147f-kserve-798f55f75-4gz47\" (UID: \"d50edaa2-d402-4678-8d8b-e265ab2c65ed\") " pod="ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47" Oct 30 07:54:04.772376 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.772323 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d50edaa2-d402-4678-8d8b-e265ab2c65ed-home\") pod \"test-model-51147f-kserve-798f55f75-4gz47\" (UID: \"d50edaa2-d402-4678-8d8b-e265ab2c65ed\") " pod="ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47" Oct 30 07:54:04.772376 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.772355 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d50edaa2-d402-4678-8d8b-e265ab2c65ed-tls-certs\") pod \"test-model-51147f-kserve-798f55f75-4gz47\" (UID: \"d50edaa2-d402-4678-8d8b-e265ab2c65ed\") " pod="ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47" Oct 30 07:54:04.772376 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.772362 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d50edaa2-d402-4678-8d8b-e265ab2c65ed-model-cache\") pod \"test-model-51147f-kserve-798f55f75-4gz47\" (UID: \"d50edaa2-d402-4678-8d8b-e265ab2c65ed\") " pod="ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47" Oct 30 07:54:04.772659 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.772633 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d50edaa2-d402-4678-8d8b-e265ab2c65ed-tmp-dir\") pod \"test-model-51147f-kserve-798f55f75-4gz47\" (UID: \"d50edaa2-d402-4678-8d8b-e265ab2c65ed\") " pod="ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47" Oct 30 07:54:04.774411 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.774387 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d50edaa2-d402-4678-8d8b-e265ab2c65ed-dshm\") pod \"test-model-51147f-kserve-798f55f75-4gz47\" (UID: \"d50edaa2-d402-4678-8d8b-e265ab2c65ed\") " pod="ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47" Oct 30 07:54:04.774755 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.774732 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d50edaa2-d402-4678-8d8b-e265ab2c65ed-tls-certs\") pod \"test-model-51147f-kserve-798f55f75-4gz47\" (UID: \"d50edaa2-d402-4678-8d8b-e265ab2c65ed\") " pod="ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47" Oct 30 07:54:04.834644 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.827759 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvsrc\" (UniqueName: \"kubernetes.io/projected/d50edaa2-d402-4678-8d8b-e265ab2c65ed-kube-api-access-rvsrc\") pod \"test-model-51147f-kserve-798f55f75-4gz47\" (UID: \"d50edaa2-d402-4678-8d8b-e265ab2c65ed\") " pod="ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47" Oct 30 07:54:04.873608 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:04.871906 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47" Oct 30 07:54:05.316085 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:54:05.316047 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd50edaa2_d402_4678_8d8b_e265ab2c65ed.slice/crio-0615c8116232bb282ac408c8452e097e263a647408a3d07e8687a9649e16b461 WatchSource:0}: Error finding container 0615c8116232bb282ac408c8452e097e263a647408a3d07e8687a9649e16b461: Status 404 returned error can't find the container with id 0615c8116232bb282ac408c8452e097e263a647408a3d07e8687a9649e16b461 Oct 30 07:54:05.316691 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:05.316661 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47"] Oct 30 07:54:05.613449 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:05.613333 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47" event={"ID":"d50edaa2-d402-4678-8d8b-e265ab2c65ed","Type":"ContainerStarted","Data":"6d9760f9b4689a3ba7ce5642b0cf47bf6b03536cfee538509aec6fb0106462fe"} Oct 30 07:54:05.613449 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:05.613385 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47" event={"ID":"d50edaa2-d402-4678-8d8b-e265ab2c65ed","Type":"ContainerStarted","Data":"0615c8116232bb282ac408c8452e097e263a647408a3d07e8687a9649e16b461"} Oct 30 07:54:05.658759 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:05.658700 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47" podStartSLOduration=1.6586829349999999 podStartE2EDuration="1.658682935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:54:05.658252559 +0000 UTC m=+3452.001300544" watchObservedRunningTime="2026-10-30 07:54:05.658682935 +0000 UTC m=+3452.001730913" Oct 30 07:54:05.872779 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:05.872700 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47" Oct 30 07:54:08.480709 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:08.480668 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd"] Oct 30 07:54:08.486544 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:08.486518 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd" Oct 30 07:54:08.506890 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:08.506850 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd"] Oct 30 07:54:08.513611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:08.513361 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-268576\"/\"openshift-service-ca.crt\"" Oct 30 07:54:08.513611 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:08.513489 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-268576\"/\"default-dockercfg-9khkg\"" Oct 30 07:54:08.519593 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:08.519545 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-268576\"/\"test-model-268576-kserve-self-signed-certs\"" Oct 30 07:54:08.585201 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:08.585163 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbwxx\" (UniqueName: \"kubernetes.io/projected/6f02b02a-47df-4c5b-a91d-36d9961e0676-kube-api-access-tbwxx\") pod \"test-model-268576-kserve-78c6b97b75-jm5dd\" (UID: \"6f02b02a-47df-4c5b-a91d-36d9961e0676\") " pod="ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd" Oct 30 07:54:08.585377 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:08.585230 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f02b02a-47df-4c5b-a91d-36d9961e0676-tmp-dir\") pod \"test-model-268576-kserve-78c6b97b75-jm5dd\" (UID: \"6f02b02a-47df-4c5b-a91d-36d9961e0676\") " pod="ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd" Oct 30 07:54:08.585377 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:08.585350 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f02b02a-47df-4c5b-a91d-36d9961e0676-home\") pod \"test-model-268576-kserve-78c6b97b75-jm5dd\" (UID: \"6f02b02a-47df-4c5b-a91d-36d9961e0676\") " pod="ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd" Oct 30 07:54:08.585475 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:08.585450 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f02b02a-47df-4c5b-a91d-36d9961e0676-dshm\") pod \"test-model-268576-kserve-78c6b97b75-jm5dd\" (UID: \"6f02b02a-47df-4c5b-a91d-36d9961e0676\") " pod="ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd" Oct 30 07:54:08.585520 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:08.585474 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f02b02a-47df-4c5b-a91d-36d9961e0676-tls-certs\") pod \"test-model-268576-kserve-78c6b97b75-jm5dd\" (UID: \"6f02b02a-47df-4c5b-a91d-36d9961e0676\") " pod="ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd" Oct 30 07:54:08.585520 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:08.585515 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f02b02a-47df-4c5b-a91d-36d9961e0676-model-cache\") pod \"test-model-268576-kserve-78c6b97b75-jm5dd\" (UID: \"6f02b02a-47df-4c5b-a91d-36d9961e0676\") " pod="ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd" Oct 30 07:54:08.603560 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:08.603535 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-268576\"/\"kube-root-ca.crt\"" Oct 30 07:54:08.686863 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:08.686817 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f02b02a-47df-4c5b-a91d-36d9961e0676-dshm\") pod \"test-model-268576-kserve-78c6b97b75-jm5dd\" (UID: \"6f02b02a-47df-4c5b-a91d-36d9961e0676\") " pod="ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd" Oct 30 07:54:08.687079 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:08.686874 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f02b02a-47df-4c5b-a91d-36d9961e0676-tls-certs\") pod \"test-model-268576-kserve-78c6b97b75-jm5dd\" (UID: \"6f02b02a-47df-4c5b-a91d-36d9961e0676\") " pod="ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd" Oct 30 07:54:08.687079 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:08.686935 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f02b02a-47df-4c5b-a91d-36d9961e0676-model-cache\") pod \"test-model-268576-kserve-78c6b97b75-jm5dd\" (UID: \"6f02b02a-47df-4c5b-a91d-36d9961e0676\") " pod="ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd" Oct 30 07:54:08.687223 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:08.687207 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbwxx\" (UniqueName: \"kubernetes.io/projected/6f02b02a-47df-4c5b-a91d-36d9961e0676-kube-api-access-tbwxx\") pod \"test-model-268576-kserve-78c6b97b75-jm5dd\" (UID: \"6f02b02a-47df-4c5b-a91d-36d9961e0676\") " pod="ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd" Oct 30 07:54:08.687307 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:08.687292 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f02b02a-47df-4c5b-a91d-36d9961e0676-tmp-dir\") pod \"test-model-268576-kserve-78c6b97b75-jm5dd\" (UID: \"6f02b02a-47df-4c5b-a91d-36d9961e0676\") " pod="ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd" Oct 30 07:54:08.687373 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:08.687356 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f02b02a-47df-4c5b-a91d-36d9961e0676-home\") pod \"test-model-268576-kserve-78c6b97b75-jm5dd\" (UID: \"6f02b02a-47df-4c5b-a91d-36d9961e0676\") " pod="ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd" Oct 30 07:54:08.687438 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:08.687358 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f02b02a-47df-4c5b-a91d-36d9961e0676-model-cache\") pod \"test-model-268576-kserve-78c6b97b75-jm5dd\" (UID: \"6f02b02a-47df-4c5b-a91d-36d9961e0676\") " pod="ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd" Oct 30 07:54:08.687731 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:08.687708 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f02b02a-47df-4c5b-a91d-36d9961e0676-tmp-dir\") pod \"test-model-268576-kserve-78c6b97b75-jm5dd\" (UID: \"6f02b02a-47df-4c5b-a91d-36d9961e0676\") " pod="ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd" Oct 30 07:54:08.687731 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:08.687722 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f02b02a-47df-4c5b-a91d-36d9961e0676-home\") pod \"test-model-268576-kserve-78c6b97b75-jm5dd\" (UID: \"6f02b02a-47df-4c5b-a91d-36d9961e0676\") " pod="ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd" Oct 30 07:54:08.689381 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:08.689363 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f02b02a-47df-4c5b-a91d-36d9961e0676-dshm\") pod \"test-model-268576-kserve-78c6b97b75-jm5dd\" (UID: \"6f02b02a-47df-4c5b-a91d-36d9961e0676\") " pod="ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd" Oct 30 07:54:08.689475 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:08.689457 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f02b02a-47df-4c5b-a91d-36d9961e0676-tls-certs\") pod \"test-model-268576-kserve-78c6b97b75-jm5dd\" (UID: \"6f02b02a-47df-4c5b-a91d-36d9961e0676\") " pod="ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd" Oct 30 07:54:08.740426 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:08.740337 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbwxx\" (UniqueName: \"kubernetes.io/projected/6f02b02a-47df-4c5b-a91d-36d9961e0676-kube-api-access-tbwxx\") pod \"test-model-268576-kserve-78c6b97b75-jm5dd\" (UID: \"6f02b02a-47df-4c5b-a91d-36d9961e0676\") " pod="ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd" Oct 30 07:54:08.815596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:08.811778 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd" Oct 30 07:54:09.251027 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:54:09.250991 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f02b02a_47df_4c5b_a91d_36d9961e0676.slice/crio-e4bc513e529a033ba4efae93a128615fae1caab6bb3356947ad353ffa83797b7 WatchSource:0}: Error finding container e4bc513e529a033ba4efae93a128615fae1caab6bb3356947ad353ffa83797b7: Status 404 returned error can't find the container with id e4bc513e529a033ba4efae93a128615fae1caab6bb3356947ad353ffa83797b7 Oct 30 07:54:09.252472 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:09.252443 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd"] Oct 30 07:54:09.654850 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:09.654808 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd" event={"ID":"6f02b02a-47df-4c5b-a91d-36d9961e0676","Type":"ContainerStarted","Data":"d61a68c952c08b4cd075408ada86dec965712d3d75776fffeb2b25e1786103aa"} Oct 30 07:54:09.655351 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:09.654865 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd" event={"ID":"6f02b02a-47df-4c5b-a91d-36d9961e0676","Type":"ContainerStarted","Data":"e4bc513e529a033ba4efae93a128615fae1caab6bb3356947ad353ffa83797b7"} Oct 30 07:54:09.705764 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:09.705703 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd" podStartSLOduration=1.705685596 podStartE2EDuration="1.705685596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:54:09.698380945 +0000 UTC m=+3456.041428935" watchObservedRunningTime="2026-10-30 07:54:09.705685596 +0000 UTC m=+3456.048733564" Oct 30 07:54:09.821602 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:09.816092 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd" Oct 30 07:54:14.872433 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:14.872394 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47" Oct 30 07:54:14.885936 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:14.885907 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47" Oct 30 07:54:15.733869 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:15.733778 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47" Oct 30 07:54:18.812957 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:18.812921 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd" Oct 30 07:54:18.825783 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:18.825753 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd" Oct 30 07:54:19.779086 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:19.779050 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd" Oct 30 07:54:23.029284 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:54:23.029207 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:54:30.433294 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:30.433248 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh"] Oct 30 07:54:30.434205 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:30.434170 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" podUID="32fb4483-9210-4a96-8296-9354fa752449" containerName="istio-proxy" containerID="cri-o://3e57c429da62714aab3b0cd76beb27e04afadded253c74b6313c6855b527dfba" gracePeriod=30 Oct 30 07:54:30.448706 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:30.447711 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29"] Oct 30 07:54:30.449195 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:30.449157 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" podUID="3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc" containerName="istio-proxy" containerID="cri-o://95fc409e40a29891a0a7ba98ea868b4c640d4b878f637ed1fca814997fc5e690" gracePeriod=30 Oct 30 07:54:35.711504 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:35.711421 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn"] Oct 30 07:54:35.712136 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:35.712108 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" podUID="30bfffda-a440-4395-a471-3f302c3a0979" containerName="istio-proxy" containerID="cri-o://0596e5544f9f4d86f8e8aa903963fe9a59257653f68f56341d7aeab037352898" gracePeriod=30 Oct 30 07:54:36.016659 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.014565 4951 generic.go:356] "Generic (PLEG): container finished" podID="3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc" containerID="95fc409e40a29891a0a7ba98ea868b4c640d4b878f637ed1fca814997fc5e690" exitCode=0 Oct 30 07:54:36.016659 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.014869 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" event={"ID":"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc","Type":"ContainerDied","Data":"95fc409e40a29891a0a7ba98ea868b4c640d4b878f637ed1fca814997fc5e690"} Oct 30 07:54:36.025853 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.025678 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:54:36.028649 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.026803 4951 generic.go:356] "Generic (PLEG): container finished" podID="32fb4483-9210-4a96-8296-9354fa752449" containerID="3e57c429da62714aab3b0cd76beb27e04afadded253c74b6313c6855b527dfba" exitCode=0 Oct 30 07:54:36.028649 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.026999 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" event={"ID":"32fb4483-9210-4a96-8296-9354fa752449","Type":"ContainerDied","Data":"3e57c429da62714aab3b0cd76beb27e04afadded253c74b6313c6855b527dfba"} Oct 30 07:54:36.028649 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.027048 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" event={"ID":"32fb4483-9210-4a96-8296-9354fa752449","Type":"ContainerDied","Data":"364b63718576bd6dc3c68fc5281f7ea21c2b6c501fcffb3f250b59259f0da11c"} Oct 30 07:54:36.028649 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.027201 4951 scope.go:122] "RemoveContainer" containerID="3e57c429da62714aab3b0cd76beb27e04afadded253c74b6313c6855b527dfba" Oct 30 07:54:36.030843 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.030820 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:54:36.042995 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.042921 4951 scope.go:122] "RemoveContainer" containerID="3e57c429da62714aab3b0cd76beb27e04afadded253c74b6313c6855b527dfba" Oct 30 07:54:36.043254 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:54:36.043225 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e57c429da62714aab3b0cd76beb27e04afadded253c74b6313c6855b527dfba\": container with ID starting with 3e57c429da62714aab3b0cd76beb27e04afadded253c74b6313c6855b527dfba not found: ID does not exist" containerID="3e57c429da62714aab3b0cd76beb27e04afadded253c74b6313c6855b527dfba" Oct 30 07:54:36.043333 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.043262 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e57c429da62714aab3b0cd76beb27e04afadded253c74b6313c6855b527dfba"} err="failed to get container status \"3e57c429da62714aab3b0cd76beb27e04afadded253c74b6313c6855b527dfba\": rpc error: code = NotFound desc = could not find container \"3e57c429da62714aab3b0cd76beb27e04afadded253c74b6313c6855b527dfba\": container with ID starting with 3e57c429da62714aab3b0cd76beb27e04afadded253c74b6313c6855b527dfba not found: ID does not exist" Oct 30 07:54:36.180323 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.180289 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-istio-envoy\") pod \"32fb4483-9210-4a96-8296-9354fa752449\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " Oct 30 07:54:36.180502 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.180385 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32fb4483-9210-4a96-8296-9354fa752449-istio-token\" (UniqueName: \"kubernetes.io/projected/32fb4483-9210-4a96-8296-9354fa752449-istio-token\") pod \"32fb4483-9210-4a96-8296-9354fa752449\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " Oct 30 07:54:36.180502 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.180429 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-credential-socket\") pod \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " Oct 30 07:54:36.180502 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.180472 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-kube-api-access-flvrx\" (UniqueName: \"kubernetes.io/projected/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-kube-api-access-flvrx\") pod \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " Oct 30 07:54:36.180693 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.180507 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32fb4483-9210-4a96-8296-9354fa752449-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32fb4483-9210-4a96-8296-9354fa752449-istiod-ca-cert\") pod \"32fb4483-9210-4a96-8296-9354fa752449\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " Oct 30 07:54:36.180693 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.180664 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istio-data\") pod \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " Oct 30 07:54:36.180818 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.180698 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32fb4483-9210-4a96-8296-9354fa752449-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32fb4483-9210-4a96-8296-9354fa752449-istio-podinfo\") pod \"32fb4483-9210-4a96-8296-9354fa752449\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " Oct 30 07:54:36.180818 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.180812 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-workload-certs\") pod \"32fb4483-9210-4a96-8296-9354fa752449\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " Oct 30 07:54:36.180928 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.180848 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-credential-socket" pod "3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc" (UID: "3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:36.180928 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.180880 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istio-token\" (UniqueName: \"kubernetes.io/projected/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istio-token\") pod \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " Oct 30 07:54:36.180928 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.180911 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32fb4483-9210-4a96-8296-9354fa752449-kube-api-access-vk8k8\" (UniqueName: \"kubernetes.io/projected/32fb4483-9210-4a96-8296-9354fa752449-kube-api-access-vk8k8\") pod \"32fb4483-9210-4a96-8296-9354fa752449\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " Oct 30 07:54:36.181115 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.180972 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-credential-socket\") pod \"32fb4483-9210-4a96-8296-9354fa752449\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " Oct 30 07:54:36.181115 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.181032 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-workload-socket\") pod \"32fb4483-9210-4a96-8296-9354fa752449\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " Oct 30 07:54:36.181115 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.181072 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istio-podinfo\") pod \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " Oct 30 07:54:36.181115 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.181082 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32fb4483-9210-4a96-8296-9354fa752449-istiod-ca-cert" pod "32fb4483-9210-4a96-8296-9354fa752449" (UID: "32fb4483-9210-4a96-8296-9354fa752449"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:54:36.181115 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.181111 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-workload-certs\") pod \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " Oct 30 07:54:36.181347 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.181126 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-workload-certs" pod "32fb4483-9210-4a96-8296-9354fa752449" (UID: "32fb4483-9210-4a96-8296-9354fa752449"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:36.181347 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.181177 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istio-data" pod "3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc" (UID: "3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:36.181347 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.181197 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istiod-ca-cert\") pod \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " Oct 30 07:54:36.181347 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.181233 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istio-envoy\") pod \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " Oct 30 07:54:36.181347 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.181294 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-istio-data\") pod \"32fb4483-9210-4a96-8296-9354fa752449\" (UID: \"32fb4483-9210-4a96-8296-9354fa752449\") " Oct 30 07:54:36.181347 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.181311 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-credential-socket" pod "32fb4483-9210-4a96-8296-9354fa752449" (UID: "32fb4483-9210-4a96-8296-9354fa752449"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:36.181347 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.181337 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-workload-socket\") pod \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\" (UID: \"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc\") " Oct 30 07:54:36.182251 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.182221 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:54:36.182382 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.182256 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:54:36.182382 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.182271 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:54:36.182382 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.182286 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32fb4483-9210-4a96-8296-9354fa752449-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:54:36.182382 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.182299 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:54:36.182647 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.182525 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-workload-socket" pod "3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc" (UID: "3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:36.182748 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.182725 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-workload-certs" pod "3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc" (UID: "3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:36.183028 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.183005 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istiod-ca-cert" pod "3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc" (UID: "3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:54:36.184163 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.184131 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istio-token" pod "3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc" (UID: "3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:54:36.184398 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.184376 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-istio-data" pod "32fb4483-9210-4a96-8296-9354fa752449" (UID: "32fb4483-9210-4a96-8296-9354fa752449"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:36.184548 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.184526 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-workload-socket" pod "32fb4483-9210-4a96-8296-9354fa752449" (UID: "32fb4483-9210-4a96-8296-9354fa752449"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:36.184948 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.184911 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istio-podinfo" pod "3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc" (UID: "3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:54:36.186072 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.186051 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32fb4483-9210-4a96-8296-9354fa752449-kube-api-access-vk8k8" pod "32fb4483-9210-4a96-8296-9354fa752449" (UID: "32fb4483-9210-4a96-8296-9354fa752449"). InnerVolumeSpecName "kube-api-access-vk8k8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:54:36.187150 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.187126 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32fb4483-9210-4a96-8296-9354fa752449-istio-token" pod "32fb4483-9210-4a96-8296-9354fa752449" (UID: "32fb4483-9210-4a96-8296-9354fa752449"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:54:36.196630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.196590 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32fb4483-9210-4a96-8296-9354fa752449-istio-podinfo" pod "32fb4483-9210-4a96-8296-9354fa752449" (UID: "32fb4483-9210-4a96-8296-9354fa752449"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:54:36.196630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.196606 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-istio-envoy" pod "32fb4483-9210-4a96-8296-9354fa752449" (UID: "32fb4483-9210-4a96-8296-9354fa752449"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:36.196755 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.196649 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-kube-api-access-flvrx" pod "3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc" (UID: "3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc"). InnerVolumeSpecName "kube-api-access-flvrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:54:36.197888 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.197869 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istio-envoy" pod "3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc" (UID: "3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:36.332829 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.296381 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:54:36.332829 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.296429 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:54:36.332829 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.296445 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:54:36.332829 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.296458 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:54:36.332829 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.296471 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:54:36.332829 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.296486 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32fb4483-9210-4a96-8296-9354fa752449-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:54:36.332829 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.296499 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flvrx\" (UniqueName: \"kubernetes.io/projected/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-kube-api-access-flvrx\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:54:36.332829 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.296512 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32fb4483-9210-4a96-8296-9354fa752449-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:54:36.332829 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.296526 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:54:36.332829 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.296539 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk8k8\" (UniqueName: \"kubernetes.io/projected/32fb4483-9210-4a96-8296-9354fa752449-kube-api-access-vk8k8\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:54:36.332829 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.296555 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32fb4483-9210-4a96-8296-9354fa752449-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:54:36.332829 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.296586 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:54:36.332829 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.296600 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:54:36.734796 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:36.734692 4951 scope.go:122] "RemoveContainer" containerID="95fc409e40a29891a0a7ba98ea868b4c640d4b878f637ed1fca814997fc5e690" Oct 30 07:54:37.037264 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:37.037219 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" event={"ID":"3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc","Type":"ContainerDied","Data":"e47408096503beb3ff8fee081e8334db5b3524d852cac9ac9cff10fbc3301ccb"} Oct 30 07:54:37.037264 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:37.037249 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh" Oct 30 07:54:37.037509 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:37.037231 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29" Oct 30 07:54:37.190981 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:37.190951 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29"] Oct 30 07:54:37.197393 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:37.197366 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-cds29"] Oct 30 07:54:37.232809 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:37.232779 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh"] Oct 30 07:54:37.239125 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:37.239094 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-99nsh"] Oct 30 07:54:37.777447 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:37.777413 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32fb4483-9210-4a96-8296-9354fa752449" path="/var/lib/kubelet/pods/32fb4483-9210-4a96-8296-9354fa752449/volumes" Oct 30 07:54:37.778053 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:37.778037 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc" path="/var/lib/kubelet/pods/3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc/volumes" Oct 30 07:54:41.082685 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:41.082652 4951 generic.go:356] "Generic (PLEG): container finished" podID="30bfffda-a440-4395-a471-3f302c3a0979" containerID="0596e5544f9f4d86f8e8aa903963fe9a59257653f68f56341d7aeab037352898" exitCode=0 Oct 30 07:54:41.083141 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:41.082722 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" event={"ID":"30bfffda-a440-4395-a471-3f302c3a0979","Type":"ContainerDied","Data":"0596e5544f9f4d86f8e8aa903963fe9a59257653f68f56341d7aeab037352898"} Oct 30 07:54:41.083141 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:41.082804 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" event={"ID":"30bfffda-a440-4395-a471-3f302c3a0979","Type":"ContainerDied","Data":"04d0533bb4f34c4a3ce60bee605aba56981ef4744ff460a417eb0fca8d901946"} Oct 30 07:54:41.083141 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:41.082825 4951 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04d0533bb4f34c4a3ce60bee605aba56981ef4744ff460a417eb0fca8d901946" Oct 30 07:54:41.105439 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:41.105417 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:54:41.233334 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:41.233257 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30bfffda-a440-4395-a471-3f302c3a0979-kube-api-access-f44rk\" (UniqueName: \"kubernetes.io/projected/30bfffda-a440-4395-a471-3f302c3a0979-kube-api-access-f44rk\") pod \"30bfffda-a440-4395-a471-3f302c3a0979\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " Oct 30 07:54:41.233334 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:41.233315 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30bfffda-a440-4395-a471-3f302c3a0979-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30bfffda-a440-4395-a471-3f302c3a0979-istio-podinfo\") pod \"30bfffda-a440-4395-a471-3f302c3a0979\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " Oct 30 07:54:41.233695 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:41.233358 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30bfffda-a440-4395-a471-3f302c3a0979-istio-token\" (UniqueName: \"kubernetes.io/projected/30bfffda-a440-4395-a471-3f302c3a0979-istio-token\") pod \"30bfffda-a440-4395-a471-3f302c3a0979\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " Oct 30 07:54:41.233695 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:41.233487 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-istio-data\") pod \"30bfffda-a440-4395-a471-3f302c3a0979\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " Oct 30 07:54:41.233695 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:41.233683 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-workload-certs\") pod \"30bfffda-a440-4395-a471-3f302c3a0979\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " Oct 30 07:54:41.235059 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:41.233754 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-credential-socket\") pod \"30bfffda-a440-4395-a471-3f302c3a0979\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " Oct 30 07:54:41.235059 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:41.233796 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-workload-socket\") pod \"30bfffda-a440-4395-a471-3f302c3a0979\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " Oct 30 07:54:41.235059 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:41.233919 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30bfffda-a440-4395-a471-3f302c3a0979-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30bfffda-a440-4395-a471-3f302c3a0979-istiod-ca-cert\") pod \"30bfffda-a440-4395-a471-3f302c3a0979\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " Oct 30 07:54:41.235059 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:41.233957 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-istio-envoy\") pod \"30bfffda-a440-4395-a471-3f302c3a0979\" (UID: \"30bfffda-a440-4395-a471-3f302c3a0979\") " Oct 30 07:54:41.236167 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:41.235974 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-credential-socket" pod "30bfffda-a440-4395-a471-3f302c3a0979" (UID: "30bfffda-a440-4395-a471-3f302c3a0979"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:41.236639 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:41.236342 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-workload-certs" pod "30bfffda-a440-4395-a471-3f302c3a0979" (UID: "30bfffda-a440-4395-a471-3f302c3a0979"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:41.237519 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:41.237482 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30bfffda-a440-4395-a471-3f302c3a0979-istiod-ca-cert" pod "30bfffda-a440-4395-a471-3f302c3a0979" (UID: "30bfffda-a440-4395-a471-3f302c3a0979"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:54:41.237930 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:41.237903 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-workload-socket" pod "30bfffda-a440-4395-a471-3f302c3a0979" (UID: "30bfffda-a440-4395-a471-3f302c3a0979"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:41.238189 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:41.238164 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-istio-data" pod "30bfffda-a440-4395-a471-3f302c3a0979" (UID: "30bfffda-a440-4395-a471-3f302c3a0979"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:41.238262 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:41.238250 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-istio-envoy" pod "30bfffda-a440-4395-a471-3f302c3a0979" (UID: "30bfffda-a440-4395-a471-3f302c3a0979"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:41.248431 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:41.248384 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30bfffda-a440-4395-a471-3f302c3a0979-istio-podinfo" pod "30bfffda-a440-4395-a471-3f302c3a0979" (UID: "30bfffda-a440-4395-a471-3f302c3a0979"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:54:41.248671 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:41.248475 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30bfffda-a440-4395-a471-3f302c3a0979-kube-api-access-f44rk" pod "30bfffda-a440-4395-a471-3f302c3a0979" (UID: "30bfffda-a440-4395-a471-3f302c3a0979"). InnerVolumeSpecName "kube-api-access-f44rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:54:41.249923 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:41.249893 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30bfffda-a440-4395-a471-3f302c3a0979-istio-token" pod "30bfffda-a440-4395-a471-3f302c3a0979" (UID: "30bfffda-a440-4395-a471-3f302c3a0979"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:54:41.336358 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:41.336281 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30bfffda-a440-4395-a471-3f302c3a0979-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:54:41.336358 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:41.336317 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:54:41.336358 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:41.336331 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f44rk\" (UniqueName: \"kubernetes.io/projected/30bfffda-a440-4395-a471-3f302c3a0979-kube-api-access-f44rk\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:54:41.336358 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:41.336346 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30bfffda-a440-4395-a471-3f302c3a0979-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:54:41.336358 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:41.336359 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30bfffda-a440-4395-a471-3f302c3a0979-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:54:41.336663 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:41.336373 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:54:41.336663 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:41.336385 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:54:41.336663 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:41.336396 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:54:41.336663 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:41.336409 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30bfffda-a440-4395-a471-3f302c3a0979-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:54:42.090637 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:42.090527 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn" Oct 30 07:54:42.165660 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:42.165628 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn"] Oct 30 07:54:42.178895 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:42.178864 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-9dqgn"] Oct 30 07:54:43.776721 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:43.776682 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30bfffda-a440-4395-a471-3f302c3a0979" path="/var/lib/kubelet/pods/30bfffda-a440-4395-a471-3f302c3a0979/volumes" Oct 30 07:54:49.532616 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:49.532559 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z"] Oct 30 07:54:49.533288 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:49.532846 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z" podUID="be63f88f-66b9-43b2-b91c-b572e8cc846d" containerName="maas-api" containerID="cri-o://d5a2d6959b8416d58c9eb0edf730dd7ac6e0904613dcceb055641dc9404a2e12" gracePeriod=30 Oct 30 07:54:50.283718 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:50.283679 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z" Oct 30 07:54:50.287212 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:50.287178 4951 generic.go:356] "Generic (PLEG): container finished" podID="be63f88f-66b9-43b2-b91c-b572e8cc846d" containerID="d5a2d6959b8416d58c9eb0edf730dd7ac6e0904613dcceb055641dc9404a2e12" exitCode=0 Oct 30 07:54:50.287419 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:50.287258 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z" event={"ID":"be63f88f-66b9-43b2-b91c-b572e8cc846d","Type":"ContainerDied","Data":"d5a2d6959b8416d58c9eb0edf730dd7ac6e0904613dcceb055641dc9404a2e12"} Oct 30 07:54:50.287419 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:50.287331 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z" event={"ID":"be63f88f-66b9-43b2-b91c-b572e8cc846d","Type":"ContainerDied","Data":"1dd8da4d1caa8932e9b79bfaa755a4145a35fc724ff5d7405b1f4f4e07f2ab66"} Oct 30 07:54:50.287419 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:50.287365 4951 scope.go:122] "RemoveContainer" containerID="d5a2d6959b8416d58c9eb0edf730dd7ac6e0904613dcceb055641dc9404a2e12" Oct 30 07:54:50.308388 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:50.308356 4951 scope.go:122] "RemoveContainer" containerID="d5a2d6959b8416d58c9eb0edf730dd7ac6e0904613dcceb055641dc9404a2e12" Oct 30 07:54:50.308778 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:54:50.308729 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5a2d6959b8416d58c9eb0edf730dd7ac6e0904613dcceb055641dc9404a2e12\": container with ID starting with d5a2d6959b8416d58c9eb0edf730dd7ac6e0904613dcceb055641dc9404a2e12 not found: ID does not exist" containerID="d5a2d6959b8416d58c9eb0edf730dd7ac6e0904613dcceb055641dc9404a2e12" Oct 30 07:54:50.308876 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:50.308770 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5a2d6959b8416d58c9eb0edf730dd7ac6e0904613dcceb055641dc9404a2e12"} err="failed to get container status \"d5a2d6959b8416d58c9eb0edf730dd7ac6e0904613dcceb055641dc9404a2e12\": rpc error: code = NotFound desc = could not find container \"d5a2d6959b8416d58c9eb0edf730dd7ac6e0904613dcceb055641dc9404a2e12\": container with ID starting with d5a2d6959b8416d58c9eb0edf730dd7ac6e0904613dcceb055641dc9404a2e12 not found: ID does not exist" Oct 30 07:54:50.496342 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:50.481696 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be63f88f-66b9-43b2-b91c-b572e8cc846d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be63f88f-66b9-43b2-b91c-b572e8cc846d-openshift-service-ca\") pod \"be63f88f-66b9-43b2-b91c-b572e8cc846d\" (UID: \"be63f88f-66b9-43b2-b91c-b572e8cc846d\") " Oct 30 07:54:50.496342 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:50.481988 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be63f88f-66b9-43b2-b91c-b572e8cc846d-kube-api-access-xwp2v\" (UniqueName: \"kubernetes.io/projected/be63f88f-66b9-43b2-b91c-b572e8cc846d-kube-api-access-xwp2v\") pod \"be63f88f-66b9-43b2-b91c-b572e8cc846d\" (UID: \"be63f88f-66b9-43b2-b91c-b572e8cc846d\") " Oct 30 07:54:50.496342 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:50.482099 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be63f88f-66b9-43b2-b91c-b572e8cc846d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/be63f88f-66b9-43b2-b91c-b572e8cc846d-metrics-certs\") pod \"be63f88f-66b9-43b2-b91c-b572e8cc846d\" (UID: \"be63f88f-66b9-43b2-b91c-b572e8cc846d\") " Oct 30 07:54:50.496342 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:50.482190 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be63f88f-66b9-43b2-b91c-b572e8cc846d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be63f88f-66b9-43b2-b91c-b572e8cc846d-maas-api-tls\") pod \"be63f88f-66b9-43b2-b91c-b572e8cc846d\" (UID: \"be63f88f-66b9-43b2-b91c-b572e8cc846d\") " Oct 30 07:54:50.496342 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:50.491092 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be63f88f-66b9-43b2-b91c-b572e8cc846d-openshift-service-ca" pod "be63f88f-66b9-43b2-b91c-b572e8cc846d" (UID: "be63f88f-66b9-43b2-b91c-b572e8cc846d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:54:50.522363 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:50.522316 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be63f88f-66b9-43b2-b91c-b572e8cc846d-maas-api-tls" pod "be63f88f-66b9-43b2-b91c-b572e8cc846d" (UID: "be63f88f-66b9-43b2-b91c-b572e8cc846d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:54:50.523016 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:50.522985 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be63f88f-66b9-43b2-b91c-b572e8cc846d-kube-api-access-xwp2v" pod "be63f88f-66b9-43b2-b91c-b572e8cc846d" (UID: "be63f88f-66b9-43b2-b91c-b572e8cc846d"). InnerVolumeSpecName "kube-api-access-xwp2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:54:50.524018 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:50.523992 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be63f88f-66b9-43b2-b91c-b572e8cc846d-metrics-certs" pod "be63f88f-66b9-43b2-b91c-b572e8cc846d" (UID: "be63f88f-66b9-43b2-b91c-b572e8cc846d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:54:50.585113 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:50.585082 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwp2v\" (UniqueName: \"kubernetes.io/projected/be63f88f-66b9-43b2-b91c-b572e8cc846d-kube-api-access-xwp2v\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:54:50.585113 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:50.585111 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be63f88f-66b9-43b2-b91c-b572e8cc846d-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:54:50.585516 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:50.585122 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be63f88f-66b9-43b2-b91c-b572e8cc846d-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:54:50.585516 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:50.585130 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be63f88f-66b9-43b2-b91c-b572e8cc846d-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:54:51.296692 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:51.296660 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z" Oct 30 07:54:51.356121 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:51.356087 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z"] Oct 30 07:54:51.372119 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:51.372069 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-63f24b-7d698fdb6b-7fr7z"] Oct 30 07:54:51.775468 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:54:51.775425 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be63f88f-66b9-43b2-b91c-b572e8cc846d" path="/var/lib/kubelet/pods/be63f88f-66b9-43b2-b91c-b572e8cc846d/volumes" Oct 30 07:54:53.028157 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:54:53.028121 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:55:17.376124 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.376085 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv"] Oct 30 07:55:17.377477 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.377453 4951 cpu_manager.go:414] "containerMap: removing container" podUID="30bfffda-a440-4395-a471-3f302c3a0979" containerName="istio-proxy" Oct 30 07:55:17.377584 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.377479 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30bfffda-a440-4395-a471-3f302c3a0979" containerName="istio-proxy" Oct 30 07:55:17.377584 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.377518 4951 cpu_manager.go:414] "containerMap: removing container" podUID="be63f88f-66b9-43b2-b91c-b572e8cc846d" containerName="maas-api" Oct 30 07:55:17.377584 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.377526 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be63f88f-66b9-43b2-b91c-b572e8cc846d" containerName="maas-api" Oct 30 07:55:17.377584 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.377548 4951 cpu_manager.go:414] "containerMap: removing container" podUID="3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc" containerName="istio-proxy" Oct 30 07:55:17.377584 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.377553 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc" containerName="istio-proxy" Oct 30 07:55:17.377584 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.377561 4951 cpu_manager.go:414] "containerMap: removing container" podUID="32fb4483-9210-4a96-8296-9354fa752449" containerName="istio-proxy" Oct 30 07:55:17.377584 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.377565 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32fb4483-9210-4a96-8296-9354fa752449" containerName="istio-proxy" Oct 30 07:55:17.377815 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.377745 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="30bfffda-a440-4395-a471-3f302c3a0979" containerName="istio-proxy" Oct 30 07:55:17.377815 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.377759 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="be63f88f-66b9-43b2-b91c-b572e8cc846d" containerName="maas-api" Oct 30 07:55:17.377815 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.377772 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a75b3c1-7b1a-47aa-ad25-3d16a3e5babc" containerName="istio-proxy" Oct 30 07:55:17.377815 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.377781 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="32fb4483-9210-4a96-8296-9354fa752449" containerName="istio-proxy" Oct 30 07:55:17.381855 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.381836 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 07:55:17.400025 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.399993 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-352fb3-openshift-default-dockercfg-t9ggs\"" Oct 30 07:55:17.418265 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.418220 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv"] Oct 30 07:55:17.455013 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.454959 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-workload-socket\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 07:55:17.455241 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.455067 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istio-envoy\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 07:55:17.455241 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.455095 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istio-podinfo\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 07:55:17.455461 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.455167 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istiod-ca-cert\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 07:55:17.455461 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.455362 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istio-data\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 07:55:17.455461 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.455394 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g9rh\" (UniqueName: \"kubernetes.io/projected/90e2dc87-6736-4b39-acfe-8db6ee096cd5-kube-api-access-2g9rh\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 07:55:17.455676 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.455630 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istio-token\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 07:55:17.455676 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.455668 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-workload-certs\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 07:55:17.455868 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.455755 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-credential-socket\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 07:55:17.565599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.564148 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istio-token\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 07:55:17.565599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.564199 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-workload-certs\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 07:55:17.565599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.564279 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-credential-socket\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 07:55:17.565599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.564483 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-workload-socket\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 07:55:17.565599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.564619 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istio-envoy\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 07:55:17.565599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.564651 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istio-podinfo\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 07:55:17.565599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.564732 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istiod-ca-cert\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 07:55:17.565599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.564773 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istio-data\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 07:55:17.565599 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.564810 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g9rh\" (UniqueName: \"kubernetes.io/projected/90e2dc87-6736-4b39-acfe-8db6ee096cd5-kube-api-access-2g9rh\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 07:55:17.580371 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.578512 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-workload-certs\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 07:55:17.580371 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.578826 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-workload-socket\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 07:55:17.580371 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.579103 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istio-data\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 07:55:17.581116 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.581089 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-credential-socket\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 07:55:17.590800 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.590773 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istiod-ca-cert\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 07:55:17.611117 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.602819 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g9rh\" (UniqueName: \"kubernetes.io/projected/90e2dc87-6736-4b39-acfe-8db6ee096cd5-kube-api-access-2g9rh\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 07:55:17.612882 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.612855 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istio-envoy\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 07:55:17.623464 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.623431 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istio-token\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 07:55:17.624775 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.624479 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istio-podinfo\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 07:55:17.696136 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:17.696054 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 07:55:18.024588 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:55:18.024544 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90e2dc87_6736_4b39_acfe_8db6ee096cd5.slice/crio-31e7878602784b5b4d56b24b3d991a46fca3f119bfb16ea67d8f6305198c71fa WatchSource:0}: Error finding container 31e7878602784b5b4d56b24b3d991a46fca3f119bfb16ea67d8f6305198c71fa: Status 404 returned error can't find the container with id 31e7878602784b5b4d56b24b3d991a46fca3f119bfb16ea67d8f6305198c71fa Oct 30 07:55:18.027110 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:18.027076 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv"] Oct 30 07:55:18.028511 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:18.028461 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:55:18.028642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:18.028563 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:55:18.028642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:18.028627 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:55:18.634481 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:18.634431 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" event={"ID":"90e2dc87-6736-4b39-acfe-8db6ee096cd5","Type":"ContainerStarted","Data":"be9f6e6fb74200d6194062f302eb451a724c724bdc2066aa5cfa5e0f1cb3e057"} Oct 30 07:55:18.634980 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:18.634493 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" event={"ID":"90e2dc87-6736-4b39-acfe-8db6ee096cd5","Type":"ContainerStarted","Data":"31e7878602784b5b4d56b24b3d991a46fca3f119bfb16ea67d8f6305198c71fa"} Oct 30 07:55:18.697192 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:18.697161 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 07:55:18.698924 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:18.698878 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" podStartSLOduration=1.69886517 podStartE2EDuration="1.69886517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:55:18.698644463 +0000 UTC m=+3525.041692429" watchObservedRunningTime="2026-10-30 07:55:18.69886517 +0000 UTC m=+3525.041913134" Oct 30 07:55:19.701282 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:19.699760 4951 patch_prober.go:29] interesting pod/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 30 07:55:19.701282 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:19.700851 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" podUID="90e2dc87-6736-4b39-acfe-8db6ee096cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 30 07:55:20.696739 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:20.696704 4951 patch_prober.go:29] interesting pod/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 30 07:55:20.697008 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:20.696755 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" podUID="90e2dc87-6736-4b39-acfe-8db6ee096cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 30 07:55:21.696853 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:21.696817 4951 patch_prober.go:29] interesting pod/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 30 07:55:21.697364 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:21.696878 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" podUID="90e2dc87-6736-4b39-acfe-8db6ee096cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 30 07:55:22.696558 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:22.696521 4951 patch_prober.go:29] interesting pod/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 30 07:55:22.696771 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:22.696605 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" podUID="90e2dc87-6736-4b39-acfe-8db6ee096cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 30 07:55:23.066696 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:55:23.066608 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:55:23.696357 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:23.696325 4951 patch_prober.go:29] interesting pod/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 30 07:55:23.696534 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:23.696374 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" podUID="90e2dc87-6736-4b39-acfe-8db6ee096cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 30 07:55:24.696497 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:24.696453 4951 patch_prober.go:29] interesting pod/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 30 07:55:24.696957 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:24.696516 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" podUID="90e2dc87-6736-4b39-acfe-8db6ee096cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 30 07:55:25.701622 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:25.701589 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 07:55:25.702047 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:25.702025 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 07:55:25.702772 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:25.702754 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 07:55:26.962563 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:26.962523 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4"] Oct 30 07:55:26.963045 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:26.962815 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" podUID="7771d5de-79de-4e6d-a0e8-acbacaf902af" containerName="istio-proxy" containerID="cri-o://65968c8fd7ec34701c0910e168db0218fa96979f80b32225dac154ca5e6ebadf" gracePeriod=30 Oct 30 07:55:32.280988 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.280960 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:55:32.378950 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.378911 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7771d5de-79de-4e6d-a0e8-acbacaf902af-istio-token\" (UniqueName: \"kubernetes.io/projected/7771d5de-79de-4e6d-a0e8-acbacaf902af-istio-token\") pod \"7771d5de-79de-4e6d-a0e8-acbacaf902af\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " Oct 30 07:55:32.379154 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.379003 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7771d5de-79de-4e6d-a0e8-acbacaf902af-kube-api-access-2wdqk\" (UniqueName: \"kubernetes.io/projected/7771d5de-79de-4e6d-a0e8-acbacaf902af-kube-api-access-2wdqk\") pod \"7771d5de-79de-4e6d-a0e8-acbacaf902af\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " Oct 30 07:55:32.379232 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.379162 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-workload-certs\") pod \"7771d5de-79de-4e6d-a0e8-acbacaf902af\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " Oct 30 07:55:32.379232 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.379221 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7771d5de-79de-4e6d-a0e8-acbacaf902af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7771d5de-79de-4e6d-a0e8-acbacaf902af-istiod-ca-cert\") pod \"7771d5de-79de-4e6d-a0e8-acbacaf902af\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " Oct 30 07:55:32.379405 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.379345 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-workload-socket\") pod \"7771d5de-79de-4e6d-a0e8-acbacaf902af\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " Oct 30 07:55:32.379405 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.379381 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-istio-envoy\") pod \"7771d5de-79de-4e6d-a0e8-acbacaf902af\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " Oct 30 07:55:32.379510 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.379438 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-istio-data\") pod \"7771d5de-79de-4e6d-a0e8-acbacaf902af\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " Oct 30 07:55:32.379510 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.379445 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-workload-certs" pod "7771d5de-79de-4e6d-a0e8-acbacaf902af" (UID: "7771d5de-79de-4e6d-a0e8-acbacaf902af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:55:32.379510 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.379470 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7771d5de-79de-4e6d-a0e8-acbacaf902af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7771d5de-79de-4e6d-a0e8-acbacaf902af-istio-podinfo\") pod \"7771d5de-79de-4e6d-a0e8-acbacaf902af\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " Oct 30 07:55:32.379630 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.379518 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-credential-socket\") pod \"7771d5de-79de-4e6d-a0e8-acbacaf902af\" (UID: \"7771d5de-79de-4e6d-a0e8-acbacaf902af\") " Oct 30 07:55:32.379669 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.379644 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-workload-socket" pod "7771d5de-79de-4e6d-a0e8-acbacaf902af" (UID: "7771d5de-79de-4e6d-a0e8-acbacaf902af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:55:32.379711 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.379676 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7771d5de-79de-4e6d-a0e8-acbacaf902af-istiod-ca-cert" pod "7771d5de-79de-4e6d-a0e8-acbacaf902af" (UID: "7771d5de-79de-4e6d-a0e8-acbacaf902af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:55:32.379886 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.379862 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-istio-data" pod "7771d5de-79de-4e6d-a0e8-acbacaf902af" (UID: "7771d5de-79de-4e6d-a0e8-acbacaf902af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:55:32.380110 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.380091 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-credential-socket" pod "7771d5de-79de-4e6d-a0e8-acbacaf902af" (UID: "7771d5de-79de-4e6d-a0e8-acbacaf902af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:55:32.380440 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.380418 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:55:32.380527 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.380448 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:55:32.380527 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.380463 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:55:32.380527 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.380476 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7771d5de-79de-4e6d-a0e8-acbacaf902af-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:55:32.380527 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.380488 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:55:32.381436 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.381411 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7771d5de-79de-4e6d-a0e8-acbacaf902af-istio-token" pod "7771d5de-79de-4e6d-a0e8-acbacaf902af" (UID: "7771d5de-79de-4e6d-a0e8-acbacaf902af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:55:32.381680 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.381660 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-istio-envoy" pod "7771d5de-79de-4e6d-a0e8-acbacaf902af" (UID: "7771d5de-79de-4e6d-a0e8-acbacaf902af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:55:32.381794 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.381779 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7771d5de-79de-4e6d-a0e8-acbacaf902af-istio-podinfo" pod "7771d5de-79de-4e6d-a0e8-acbacaf902af" (UID: "7771d5de-79de-4e6d-a0e8-acbacaf902af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:55:32.382690 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.382667 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7771d5de-79de-4e6d-a0e8-acbacaf902af-kube-api-access-2wdqk" pod "7771d5de-79de-4e6d-a0e8-acbacaf902af" (UID: "7771d5de-79de-4e6d-a0e8-acbacaf902af"). InnerVolumeSpecName "kube-api-access-2wdqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:55:32.482171 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.482130 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7771d5de-79de-4e6d-a0e8-acbacaf902af-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:55:32.482171 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.482165 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wdqk\" (UniqueName: \"kubernetes.io/projected/7771d5de-79de-4e6d-a0e8-acbacaf902af-kube-api-access-2wdqk\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:55:32.482171 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.482177 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7771d5de-79de-4e6d-a0e8-acbacaf902af-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:55:32.482438 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.482185 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7771d5de-79de-4e6d-a0e8-acbacaf902af-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:55:32.786027 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.785991 4951 generic.go:356] "Generic (PLEG): container finished" podID="7771d5de-79de-4e6d-a0e8-acbacaf902af" containerID="65968c8fd7ec34701c0910e168db0218fa96979f80b32225dac154ca5e6ebadf" exitCode=0 Oct 30 07:55:32.786220 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.786025 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" event={"ID":"7771d5de-79de-4e6d-a0e8-acbacaf902af","Type":"ContainerDied","Data":"65968c8fd7ec34701c0910e168db0218fa96979f80b32225dac154ca5e6ebadf"} Oct 30 07:55:32.786220 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.786059 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" Oct 30 07:55:32.786220 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.786092 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4" event={"ID":"7771d5de-79de-4e6d-a0e8-acbacaf902af","Type":"ContainerDied","Data":"fed9beabd50518f8bf56161bb0fcf83561aea32c3be5e4633eca84ea73fb54d6"} Oct 30 07:55:32.786220 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.786124 4951 scope.go:122] "RemoveContainer" containerID="65968c8fd7ec34701c0910e168db0218fa96979f80b32225dac154ca5e6ebadf" Oct 30 07:55:32.801230 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.801192 4951 scope.go:122] "RemoveContainer" containerID="e6a584ae28bee7ee12fa124066766c0857e3af029a91b49eec1f1739d75fba0f" Oct 30 07:55:32.812606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.812556 4951 scope.go:122] "RemoveContainer" containerID="65968c8fd7ec34701c0910e168db0218fa96979f80b32225dac154ca5e6ebadf" Oct 30 07:55:32.812884 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:55:32.812865 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65968c8fd7ec34701c0910e168db0218fa96979f80b32225dac154ca5e6ebadf\": container with ID starting with 65968c8fd7ec34701c0910e168db0218fa96979f80b32225dac154ca5e6ebadf not found: ID does not exist" containerID="65968c8fd7ec34701c0910e168db0218fa96979f80b32225dac154ca5e6ebadf" Oct 30 07:55:32.812962 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.812894 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65968c8fd7ec34701c0910e168db0218fa96979f80b32225dac154ca5e6ebadf"} err="failed to get container status \"65968c8fd7ec34701c0910e168db0218fa96979f80b32225dac154ca5e6ebadf\": rpc error: code = NotFound desc = could not find container \"65968c8fd7ec34701c0910e168db0218fa96979f80b32225dac154ca5e6ebadf\": container with ID starting with 65968c8fd7ec34701c0910e168db0218fa96979f80b32225dac154ca5e6ebadf not found: ID does not exist" Oct 30 07:55:32.812962 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.812912 4951 scope.go:122] "RemoveContainer" containerID="e6a584ae28bee7ee12fa124066766c0857e3af029a91b49eec1f1739d75fba0f" Oct 30 07:55:32.813146 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:55:32.813126 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6a584ae28bee7ee12fa124066766c0857e3af029a91b49eec1f1739d75fba0f\": container with ID starting with e6a584ae28bee7ee12fa124066766c0857e3af029a91b49eec1f1739d75fba0f not found: ID does not exist" containerID="e6a584ae28bee7ee12fa124066766c0857e3af029a91b49eec1f1739d75fba0f" Oct 30 07:55:32.813211 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.813157 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6a584ae28bee7ee12fa124066766c0857e3af029a91b49eec1f1739d75fba0f"} err="failed to get container status \"e6a584ae28bee7ee12fa124066766c0857e3af029a91b49eec1f1739d75fba0f\": rpc error: code = NotFound desc = could not find container \"e6a584ae28bee7ee12fa124066766c0857e3af029a91b49eec1f1739d75fba0f\": container with ID starting with e6a584ae28bee7ee12fa124066766c0857e3af029a91b49eec1f1739d75fba0f not found: ID does not exist" Oct 30 07:55:32.845723 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.845696 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4"] Oct 30 07:55:32.858606 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:32.858581 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-cv8x4"] Oct 30 07:55:33.819253 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:33.819214 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7771d5de-79de-4e6d-a0e8-acbacaf902af" path="/var/lib/kubelet/pods/7771d5de-79de-4e6d-a0e8-acbacaf902af/volumes" Oct 30 07:55:36.798594 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:36.795320 4951 scope.go:122] "RemoveContainer" containerID="0596e5544f9f4d86f8e8aa903963fe9a59257653f68f56341d7aeab037352898" Oct 30 07:55:38.093995 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:38.093960 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-61f6c2-d6f8dd8fd-tb2zd"] Oct 30 07:55:43.094438 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:43.094380 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-61f6c2-d6f8dd8fd-tb2zd" podUID="2e3d3e6e-232d-48ea-a24e-e25812154cb7" containerName="payload-processing" containerID="cri-o://b6e6c2ae3e038a6bb99b8e2573b28a109b819f5be045748690b9cb55c59006cb" gracePeriod=25 Oct 30 07:55:52.998590 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:55:52.998529 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-61f6c2-d6f8dd8fd-tb2zd" podUID="2e3d3e6e-232d-48ea-a24e-e25812154cb7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:55:53.025743 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:55:53.025703 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:56:02.998420 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:02.998380 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-61f6c2-d6f8dd8fd-tb2zd" podUID="2e3d3e6e-232d-48ea-a24e-e25812154cb7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:56:03.216145 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:03.216116 4951 generic.go:356] "Generic (PLEG): container finished" podID="2e3d3e6e-232d-48ea-a24e-e25812154cb7" containerID="b6e6c2ae3e038a6bb99b8e2573b28a109b819f5be045748690b9cb55c59006cb" exitCode=0 Oct 30 07:56:03.216262 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:03.216176 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-61f6c2-d6f8dd8fd-tb2zd" event={"ID":"2e3d3e6e-232d-48ea-a24e-e25812154cb7","Type":"ContainerDied","Data":"b6e6c2ae3e038a6bb99b8e2573b28a109b819f5be045748690b9cb55c59006cb"} Oct 30 07:56:03.489792 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:03.489761 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-61f6c2-d6f8dd8fd-tb2zd" Oct 30 07:56:03.659164 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:03.659072 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e3d3e6e-232d-48ea-a24e-e25812154cb7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3d3e6e-232d-48ea-a24e-e25812154cb7-plugins-config-volume\") pod \"2e3d3e6e-232d-48ea-a24e-e25812154cb7\" (UID: \"2e3d3e6e-232d-48ea-a24e-e25812154cb7\") " Oct 30 07:56:03.659343 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:03.659302 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e3d3e6e-232d-48ea-a24e-e25812154cb7-kube-api-access-r9v4w\" (UniqueName: \"kubernetes.io/projected/2e3d3e6e-232d-48ea-a24e-e25812154cb7-kube-api-access-r9v4w\") pod \"2e3d3e6e-232d-48ea-a24e-e25812154cb7\" (UID: \"2e3d3e6e-232d-48ea-a24e-e25812154cb7\") " Oct 30 07:56:03.659549 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:03.659520 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e3d3e6e-232d-48ea-a24e-e25812154cb7-plugins-config-volume" pod "2e3d3e6e-232d-48ea-a24e-e25812154cb7" (UID: "2e3d3e6e-232d-48ea-a24e-e25812154cb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:56:03.659952 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:03.659928 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3d3e6e-232d-48ea-a24e-e25812154cb7-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:56:03.661519 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:03.661495 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3d3e6e-232d-48ea-a24e-e25812154cb7-kube-api-access-r9v4w" pod "2e3d3e6e-232d-48ea-a24e-e25812154cb7" (UID: "2e3d3e6e-232d-48ea-a24e-e25812154cb7"). InnerVolumeSpecName "kube-api-access-r9v4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:56:03.761157 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:03.761123 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9v4w\" (UniqueName: \"kubernetes.io/projected/2e3d3e6e-232d-48ea-a24e-e25812154cb7-kube-api-access-r9v4w\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:56:04.224954 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.224923 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-61f6c2-d6f8dd8fd-tb2zd" Oct 30 07:56:04.225424 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.224946 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-61f6c2-d6f8dd8fd-tb2zd" event={"ID":"2e3d3e6e-232d-48ea-a24e-e25812154cb7","Type":"ContainerDied","Data":"1d59aadf7936e64afd7a935db5c1ee0e4c626c3bb20b7a375fe8f41fb825cd0c"} Oct 30 07:56:04.225424 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.225013 4951 scope.go:122] "RemoveContainer" containerID="b6e6c2ae3e038a6bb99b8e2573b28a109b819f5be045748690b9cb55c59006cb" Oct 30 07:56:04.284774 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.284740 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-61f6c2-d6f8dd8fd-tb2zd"] Oct 30 07:56:04.296642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.296614 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-61f6c2-d6f8dd8fd-tb2zd"] Oct 30 07:56:04.518830 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.518797 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b"] Oct 30 07:56:04.519729 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.519705 4951 cpu_manager.go:414] "containerMap: removing container" podUID="7771d5de-79de-4e6d-a0e8-acbacaf902af" containerName="istio-proxy" Oct 30 07:56:04.519729 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.519725 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7771d5de-79de-4e6d-a0e8-acbacaf902af" containerName="istio-proxy" Oct 30 07:56:04.520034 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.519760 4951 cpu_manager.go:414] "containerMap: removing container" podUID="2e3d3e6e-232d-48ea-a24e-e25812154cb7" containerName="payload-processing" Oct 30 07:56:04.520034 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.519766 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e3d3e6e-232d-48ea-a24e-e25812154cb7" containerName="payload-processing" Oct 30 07:56:04.520034 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.519941 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="7771d5de-79de-4e6d-a0e8-acbacaf902af" containerName="istio-proxy" Oct 30 07:56:04.520034 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.519958 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e3d3e6e-232d-48ea-a24e-e25812154cb7" containerName="payload-processing" Oct 30 07:56:04.525642 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.525614 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 07:56:04.563857 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.563823 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b"] Oct 30 07:56:04.675203 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.675162 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-xx84b\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 07:56:04.675403 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.675295 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-xx84b\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 07:56:04.675464 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.675446 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/705d9fa4-a31c-4fc9-b27b-48874576f32a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-xx84b\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 07:56:04.675512 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.675495 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-xx84b\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 07:56:04.675549 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.675536 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4hhx\" (UniqueName: \"kubernetes.io/projected/705d9fa4-a31c-4fc9-b27b-48874576f32a-kube-api-access-f4hhx\") pod \"maas-default-gateway-openshift-default-6dcf757899-xx84b\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 07:56:04.675693 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.675676 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-xx84b\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 07:56:04.675739 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.675701 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/705d9fa4-a31c-4fc9-b27b-48874576f32a-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-xx84b\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 07:56:04.675739 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.675722 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/705d9fa4-a31c-4fc9-b27b-48874576f32a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-xx84b\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 07:56:04.675869 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.675855 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-xx84b\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 07:56:04.777218 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.777125 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-xx84b\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 07:56:04.777218 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.777184 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4hhx\" (UniqueName: \"kubernetes.io/projected/705d9fa4-a31c-4fc9-b27b-48874576f32a-kube-api-access-f4hhx\") pod \"maas-default-gateway-openshift-default-6dcf757899-xx84b\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 07:56:04.777468 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.777280 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-xx84b\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 07:56:04.777468 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.777304 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/705d9fa4-a31c-4fc9-b27b-48874576f32a-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-xx84b\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 07:56:04.777468 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.777335 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/705d9fa4-a31c-4fc9-b27b-48874576f32a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-xx84b\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 07:56:04.777468 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.777438 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-xx84b\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 07:56:04.777760 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.777516 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-xx84b\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 07:56:04.777760 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.777611 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-xx84b\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 07:56:04.777760 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.777648 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-xx84b\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 07:56:04.777924 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.777811 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/705d9fa4-a31c-4fc9-b27b-48874576f32a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-xx84b\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 07:56:04.777924 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.777906 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-xx84b\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 07:56:04.778126 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.778102 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-xx84b\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 07:56:04.778207 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.778182 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-xx84b\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 07:56:04.778474 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.778444 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/705d9fa4-a31c-4fc9-b27b-48874576f32a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-xx84b\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 07:56:04.780234 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.780213 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-xx84b\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 07:56:04.780460 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.780439 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/705d9fa4-a31c-4fc9-b27b-48874576f32a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-xx84b\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 07:56:04.803807 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.803775 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/705d9fa4-a31c-4fc9-b27b-48874576f32a-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-xx84b\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 07:56:04.803962 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.803919 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4hhx\" (UniqueName: \"kubernetes.io/projected/705d9fa4-a31c-4fc9-b27b-48874576f32a-kube-api-access-f4hhx\") pod \"maas-default-gateway-openshift-default-6dcf757899-xx84b\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 07:56:04.868961 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:04.868921 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 07:56:05.232352 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:05.232321 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b"] Oct 30 07:56:05.237505 ip-10-0-132-173 kubenswrapper[4951]: W1030 07:56:05.237436 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod705d9fa4_a31c_4fc9_b27b_48874576f32a.slice/crio-5da7996fb4e739a51d2b725b24d4d0b826fdf22ed7eed4fa2f9ccf5ba0b804c5 WatchSource:0}: Error finding container 5da7996fb4e739a51d2b725b24d4d0b826fdf22ed7eed4fa2f9ccf5ba0b804c5: Status 404 returned error can't find the container with id 5da7996fb4e739a51d2b725b24d4d0b826fdf22ed7eed4fa2f9ccf5ba0b804c5 Oct 30 07:56:05.241517 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:05.241470 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:56:05.241624 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:05.241562 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:56:05.241721 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:05.241644 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:56:05.813595 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:05.813544 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e3d3e6e-232d-48ea-a24e-e25812154cb7" path="/var/lib/kubelet/pods/2e3d3e6e-232d-48ea-a24e-e25812154cb7/volumes" Oct 30 07:56:06.373171 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:06.373131 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" event={"ID":"705d9fa4-a31c-4fc9-b27b-48874576f32a","Type":"ContainerStarted","Data":"7e20545d01605913c47db9d44daac8ea73e912bd464bd6c46e9fc78bd0293a2b"} Oct 30 07:56:06.373797 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:06.373767 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" event={"ID":"705d9fa4-a31c-4fc9-b27b-48874576f32a","Type":"ContainerStarted","Data":"5da7996fb4e739a51d2b725b24d4d0b826fdf22ed7eed4fa2f9ccf5ba0b804c5"} Oct 30 07:56:06.494598 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:06.486607 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" podStartSLOduration=2.486587391 podStartE2EDuration="2.486587391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:56:06.446019477 +0000 UTC m=+3572.789067464" watchObservedRunningTime="2026-10-30 07:56:06.486587391 +0000 UTC m=+3572.829635348" Oct 30 07:56:06.870447 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:06.870389 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 07:56:06.871759 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:06.871729 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-xx84b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Oct 30 07:56:06.871927 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:06.871794 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" podUID="705d9fa4-a31c-4fc9-b27b-48874576f32a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Oct 30 07:56:07.869939 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:07.869904 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-xx84b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Oct 30 07:56:07.870398 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:07.869977 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" podUID="705d9fa4-a31c-4fc9-b27b-48874576f32a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Oct 30 07:56:08.870267 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:08.870227 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-xx84b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Oct 30 07:56:08.870787 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:08.870284 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" podUID="705d9fa4-a31c-4fc9-b27b-48874576f32a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Oct 30 07:56:09.870070 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:09.870029 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-xx84b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Oct 30 07:56:09.870267 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:09.870099 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" podUID="705d9fa4-a31c-4fc9-b27b-48874576f32a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Oct 30 07:56:10.870210 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:10.870160 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-xx84b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Oct 30 07:56:10.870700 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:10.870244 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" podUID="705d9fa4-a31c-4fc9-b27b-48874576f32a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Oct 30 07:56:11.870027 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:11.869984 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-xx84b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Oct 30 07:56:11.870215 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:11.870091 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" podUID="705d9fa4-a31c-4fc9-b27b-48874576f32a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Oct 30 07:56:12.869532 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:12.869494 4951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-xx84b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Oct 30 07:56:12.869996 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:12.869563 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" podUID="705d9fa4-a31c-4fc9-b27b-48874576f32a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Oct 30 07:56:14.338192 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:14.338153 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 07:56:14.339269 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:14.339236 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 07:56:14.339849 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:14.339828 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 07:56:23.051615 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:56:23.051564 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:56:29.433842 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:29.433801 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx"] Oct 30 07:56:29.434757 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:29.434717 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx" podUID="e64255a8-c8c4-4ed3-b280-4d35314e746d" containerName="maas-api" containerID="cri-o://5978f86e4ff9696fba8e2640b115a35583b785cf45b52f6fa2b4f07bfcc2e1a4" gracePeriod=30 Oct 30 07:56:29.584332 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:29.584291 4951 generic.go:356] "Generic (PLEG): container finished" podID="e64255a8-c8c4-4ed3-b280-4d35314e746d" containerID="5978f86e4ff9696fba8e2640b115a35583b785cf45b52f6fa2b4f07bfcc2e1a4" exitCode=0 Oct 30 07:56:29.584524 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:29.584365 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx" event={"ID":"e64255a8-c8c4-4ed3-b280-4d35314e746d","Type":"ContainerDied","Data":"5978f86e4ff9696fba8e2640b115a35583b785cf45b52f6fa2b4f07bfcc2e1a4"} Oct 30 07:56:30.201782 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:30.201751 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx" Oct 30 07:56:30.277735 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:30.277690 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e64255a8-c8c4-4ed3-b280-4d35314e746d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e64255a8-c8c4-4ed3-b280-4d35314e746d-maas-api-tls\") pod \"e64255a8-c8c4-4ed3-b280-4d35314e746d\" (UID: \"e64255a8-c8c4-4ed3-b280-4d35314e746d\") " Oct 30 07:56:30.277949 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:30.277931 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e64255a8-c8c4-4ed3-b280-4d35314e746d-kube-api-access-8fqnj\" (UniqueName: \"kubernetes.io/projected/e64255a8-c8c4-4ed3-b280-4d35314e746d-kube-api-access-8fqnj\") pod \"e64255a8-c8c4-4ed3-b280-4d35314e746d\" (UID: \"e64255a8-c8c4-4ed3-b280-4d35314e746d\") " Oct 30 07:56:30.278059 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:30.277990 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e64255a8-c8c4-4ed3-b280-4d35314e746d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e64255a8-c8c4-4ed3-b280-4d35314e746d-metrics-certs\") pod \"e64255a8-c8c4-4ed3-b280-4d35314e746d\" (UID: \"e64255a8-c8c4-4ed3-b280-4d35314e746d\") " Oct 30 07:56:30.278222 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:30.278202 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e64255a8-c8c4-4ed3-b280-4d35314e746d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e64255a8-c8c4-4ed3-b280-4d35314e746d-openshift-service-ca\") pod \"e64255a8-c8c4-4ed3-b280-4d35314e746d\" (UID: \"e64255a8-c8c4-4ed3-b280-4d35314e746d\") " Oct 30 07:56:30.279911 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:30.279878 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e64255a8-c8c4-4ed3-b280-4d35314e746d-openshift-service-ca" pod "e64255a8-c8c4-4ed3-b280-4d35314e746d" (UID: "e64255a8-c8c4-4ed3-b280-4d35314e746d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:56:30.304339 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:30.302023 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e64255a8-c8c4-4ed3-b280-4d35314e746d-maas-api-tls" pod "e64255a8-c8c4-4ed3-b280-4d35314e746d" (UID: "e64255a8-c8c4-4ed3-b280-4d35314e746d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:56:30.312650 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:30.312618 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e64255a8-c8c4-4ed3-b280-4d35314e746d-kube-api-access-8fqnj" pod "e64255a8-c8c4-4ed3-b280-4d35314e746d" (UID: "e64255a8-c8c4-4ed3-b280-4d35314e746d"). InnerVolumeSpecName "kube-api-access-8fqnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:56:30.322034 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:30.321863 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e64255a8-c8c4-4ed3-b280-4d35314e746d-metrics-certs" pod "e64255a8-c8c4-4ed3-b280-4d35314e746d" (UID: "e64255a8-c8c4-4ed3-b280-4d35314e746d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:56:30.410022 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:30.396803 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e64255a8-c8c4-4ed3-b280-4d35314e746d-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:56:30.410022 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:30.406409 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e64255a8-c8c4-4ed3-b280-4d35314e746d-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:56:30.410022 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:30.406452 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fqnj\" (UniqueName: \"kubernetes.io/projected/e64255a8-c8c4-4ed3-b280-4d35314e746d-kube-api-access-8fqnj\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:56:30.410022 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:30.406471 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e64255a8-c8c4-4ed3-b280-4d35314e746d-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:56:30.594269 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:30.594226 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx" event={"ID":"e64255a8-c8c4-4ed3-b280-4d35314e746d","Type":"ContainerDied","Data":"d5b19ebb3a449e92d2c106463ec22bbe1a07eaca5a99520da24f14550d79da66"} Oct 30 07:56:30.594269 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:30.594253 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx" Oct 30 07:56:30.594787 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:30.594325 4951 scope.go:122] "RemoveContainer" containerID="5978f86e4ff9696fba8e2640b115a35583b785cf45b52f6fa2b4f07bfcc2e1a4" Oct 30 07:56:30.654367 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:30.654318 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx"] Oct 30 07:56:30.658674 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:30.658639 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-61f6c2-6b5c49fb75-vdzjx"] Oct 30 07:56:31.775249 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:31.775213 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e64255a8-c8c4-4ed3-b280-4d35314e746d" path="/var/lib/kubelet/pods/e64255a8-c8c4-4ed3-b280-4d35314e746d/volumes" Oct 30 07:56:34.682832 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:34.682799 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/3.log" Oct 30 07:56:34.687741 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:34.687719 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 07:56:34.698353 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:34.698329 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/3.log" Oct 30 07:56:34.703560 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:34.703536 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 07:56:37.515848 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:37.515810 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx"] Oct 30 07:56:37.516329 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:37.516090 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" podUID="0e611ef5-131d-44f9-848d-3e9ae765f733" containerName="istio-proxy" containerID="cri-o://7c1ddacc8d8bd717f0ef4408be5d4c1f66f326dd427479a5894f5a4292b56ccd" gracePeriod=30 Oct 30 07:56:38.954362 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:38.954318 4951 patch_prober.go:29] interesting pod/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:56:38.954830 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:38.954408 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" podUID="0e611ef5-131d-44f9-848d-3e9ae765f733" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:56:42.846904 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:42.846378 4951 generic.go:356] "Generic (PLEG): container finished" podID="0e611ef5-131d-44f9-848d-3e9ae765f733" containerID="7c1ddacc8d8bd717f0ef4408be5d4c1f66f326dd427479a5894f5a4292b56ccd" exitCode=0 Oct 30 07:56:42.846904 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:42.846690 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" event={"ID":"0e611ef5-131d-44f9-848d-3e9ae765f733","Type":"ContainerDied","Data":"7c1ddacc8d8bd717f0ef4408be5d4c1f66f326dd427479a5894f5a4292b56ccd"} Oct 30 07:56:42.846904 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:42.846756 4951 scope.go:122] "RemoveContainer" containerID="9048f23198076171f275a33798853ea0290b4cb69ea58e343c7180f1c21feea9" Oct 30 07:56:42.896288 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:42.896259 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:56:42.936658 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:42.936620 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-workload-certs\") pod \"0e611ef5-131d-44f9-848d-3e9ae765f733\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " Oct 30 07:56:42.936805 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:42.936670 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e611ef5-131d-44f9-848d-3e9ae765f733-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e611ef5-131d-44f9-848d-3e9ae765f733-istiod-ca-cert\") pod \"0e611ef5-131d-44f9-848d-3e9ae765f733\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " Oct 30 07:56:42.936805 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:42.936782 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e611ef5-131d-44f9-848d-3e9ae765f733-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e611ef5-131d-44f9-848d-3e9ae765f733-istio-podinfo\") pod \"0e611ef5-131d-44f9-848d-3e9ae765f733\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " Oct 30 07:56:42.936922 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:42.936879 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e611ef5-131d-44f9-848d-3e9ae765f733-istio-token\" (UniqueName: \"kubernetes.io/projected/0e611ef5-131d-44f9-848d-3e9ae765f733-istio-token\") pod \"0e611ef5-131d-44f9-848d-3e9ae765f733\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " Oct 30 07:56:42.936922 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:42.936896 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-workload-certs" pod "0e611ef5-131d-44f9-848d-3e9ae765f733" (UID: "0e611ef5-131d-44f9-848d-3e9ae765f733"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:56:42.937044 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:42.936951 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-istio-envoy\") pod \"0e611ef5-131d-44f9-848d-3e9ae765f733\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " Oct 30 07:56:42.937104 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:42.937055 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-workload-socket\") pod \"0e611ef5-131d-44f9-848d-3e9ae765f733\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " Oct 30 07:56:42.937163 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:42.937101 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-credential-socket\") pod \"0e611ef5-131d-44f9-848d-3e9ae765f733\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " Oct 30 07:56:42.937163 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:42.937128 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e611ef5-131d-44f9-848d-3e9ae765f733-istiod-ca-cert" pod "0e611ef5-131d-44f9-848d-3e9ae765f733" (UID: "0e611ef5-131d-44f9-848d-3e9ae765f733"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:56:42.937270 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:42.937241 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e611ef5-131d-44f9-848d-3e9ae765f733-kube-api-access-n6scf\" (UniqueName: \"kubernetes.io/projected/0e611ef5-131d-44f9-848d-3e9ae765f733-kube-api-access-n6scf\") pod \"0e611ef5-131d-44f9-848d-3e9ae765f733\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " Oct 30 07:56:42.937326 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:42.937291 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-istio-data\") pod \"0e611ef5-131d-44f9-848d-3e9ae765f733\" (UID: \"0e611ef5-131d-44f9-848d-3e9ae765f733\") " Oct 30 07:56:42.937761 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:42.937736 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-credential-socket" pod "0e611ef5-131d-44f9-848d-3e9ae765f733" (UID: "0e611ef5-131d-44f9-848d-3e9ae765f733"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:56:42.937894 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:42.937877 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-workload-socket" pod "0e611ef5-131d-44f9-848d-3e9ae765f733" (UID: "0e611ef5-131d-44f9-848d-3e9ae765f733"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:56:42.938066 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:42.938032 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-istio-data" pod "0e611ef5-131d-44f9-848d-3e9ae765f733" (UID: "0e611ef5-131d-44f9-848d-3e9ae765f733"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:56:42.938917 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:42.938893 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:56:42.939063 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:42.939047 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:56:42.939168 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:42.939154 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:56:42.939268 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:42.939258 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:56:42.939367 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:42.939356 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e611ef5-131d-44f9-848d-3e9ae765f733-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:56:42.939958 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:42.939901 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-istio-envoy" pod "0e611ef5-131d-44f9-848d-3e9ae765f733" (UID: "0e611ef5-131d-44f9-848d-3e9ae765f733"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:56:42.939958 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:42.939927 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e611ef5-131d-44f9-848d-3e9ae765f733-istio-podinfo" pod "0e611ef5-131d-44f9-848d-3e9ae765f733" (UID: "0e611ef5-131d-44f9-848d-3e9ae765f733"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:56:42.940213 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:42.940195 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e611ef5-131d-44f9-848d-3e9ae765f733-istio-token" pod "0e611ef5-131d-44f9-848d-3e9ae765f733" (UID: "0e611ef5-131d-44f9-848d-3e9ae765f733"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:56:42.940304 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:42.940289 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e611ef5-131d-44f9-848d-3e9ae765f733-kube-api-access-n6scf" pod "0e611ef5-131d-44f9-848d-3e9ae765f733" (UID: "0e611ef5-131d-44f9-848d-3e9ae765f733"). InnerVolumeSpecName "kube-api-access-n6scf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:56:43.040612 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:43.040553 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e611ef5-131d-44f9-848d-3e9ae765f733-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:56:43.040612 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:43.040605 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e611ef5-131d-44f9-848d-3e9ae765f733-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:56:43.040612 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:43.040615 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6scf\" (UniqueName: \"kubernetes.io/projected/0e611ef5-131d-44f9-848d-3e9ae765f733-kube-api-access-n6scf\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:56:43.040886 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:43.040625 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e611ef5-131d-44f9-848d-3e9ae765f733-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:56:43.856736 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:43.856700 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" Oct 30 07:56:43.857201 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:43.856696 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx" event={"ID":"0e611ef5-131d-44f9-848d-3e9ae765f733","Type":"ContainerDied","Data":"fe0a38bb5d819f10f162f66b412c7391ed73042d62010288e1786498ed8285dd"} Oct 30 07:56:43.857201 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:43.856893 4951 scope.go:122] "RemoveContainer" containerID="7c1ddacc8d8bd717f0ef4408be5d4c1f66f326dd427479a5894f5a4292b56ccd" Oct 30 07:56:43.964605 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:43.964563 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx"] Oct 30 07:56:43.970403 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:43.970381 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-61f6c2-openshift-default-6b5948cc8f-2p5lx"] Oct 30 07:56:45.775400 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:45.775364 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e611ef5-131d-44f9-848d-3e9ae765f733" path="/var/lib/kubelet/pods/0e611ef5-131d-44f9-848d-3e9ae765f733/volumes" Oct 30 07:56:51.050328 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:51.050293 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn"] Oct 30 07:56:51.050874 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:51.050603 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn" podUID="da8a133c-4e96-4c80-aafa-a08a1082545b" containerName="maas-api" containerID="cri-o://8be600bc8e81c2bbadb72cec0556e43218493feeee46b688099ab831a0fe43fa" gracePeriod=30 Oct 30 07:56:51.617800 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:51.617766 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn" Oct 30 07:56:51.690664 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:51.688453 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da8a133c-4e96-4c80-aafa-a08a1082545b-kube-api-access-hvsxp\" (UniqueName: \"kubernetes.io/projected/da8a133c-4e96-4c80-aafa-a08a1082545b-kube-api-access-hvsxp\") pod \"da8a133c-4e96-4c80-aafa-a08a1082545b\" (UID: \"da8a133c-4e96-4c80-aafa-a08a1082545b\") " Oct 30 07:56:51.690664 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:51.688691 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da8a133c-4e96-4c80-aafa-a08a1082545b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da8a133c-4e96-4c80-aafa-a08a1082545b-metrics-certs\") pod \"da8a133c-4e96-4c80-aafa-a08a1082545b\" (UID: \"da8a133c-4e96-4c80-aafa-a08a1082545b\") " Oct 30 07:56:51.690664 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:51.688853 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da8a133c-4e96-4c80-aafa-a08a1082545b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da8a133c-4e96-4c80-aafa-a08a1082545b-maas-api-tls\") pod \"da8a133c-4e96-4c80-aafa-a08a1082545b\" (UID: \"da8a133c-4e96-4c80-aafa-a08a1082545b\") " Oct 30 07:56:51.690664 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:51.688990 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da8a133c-4e96-4c80-aafa-a08a1082545b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da8a133c-4e96-4c80-aafa-a08a1082545b-openshift-service-ca\") pod \"da8a133c-4e96-4c80-aafa-a08a1082545b\" (UID: \"da8a133c-4e96-4c80-aafa-a08a1082545b\") " Oct 30 07:56:51.690664 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:51.690458 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da8a133c-4e96-4c80-aafa-a08a1082545b-openshift-service-ca" pod "da8a133c-4e96-4c80-aafa-a08a1082545b" (UID: "da8a133c-4e96-4c80-aafa-a08a1082545b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:56:51.730702 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:51.720717 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da8a133c-4e96-4c80-aafa-a08a1082545b-metrics-certs" pod "da8a133c-4e96-4c80-aafa-a08a1082545b" (UID: "da8a133c-4e96-4c80-aafa-a08a1082545b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:56:51.730702 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:51.721526 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da8a133c-4e96-4c80-aafa-a08a1082545b-maas-api-tls" pod "da8a133c-4e96-4c80-aafa-a08a1082545b" (UID: "da8a133c-4e96-4c80-aafa-a08a1082545b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:56:51.730702 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:51.727738 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da8a133c-4e96-4c80-aafa-a08a1082545b-kube-api-access-hvsxp" pod "da8a133c-4e96-4c80-aafa-a08a1082545b" (UID: "da8a133c-4e96-4c80-aafa-a08a1082545b"). InnerVolumeSpecName "kube-api-access-hvsxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:56:51.808661 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:51.804267 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvsxp\" (UniqueName: \"kubernetes.io/projected/da8a133c-4e96-4c80-aafa-a08a1082545b-kube-api-access-hvsxp\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:56:51.808967 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:51.808945 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da8a133c-4e96-4c80-aafa-a08a1082545b-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:56:51.809075 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:51.809064 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da8a133c-4e96-4c80-aafa-a08a1082545b-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:56:51.809159 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:51.809149 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da8a133c-4e96-4c80-aafa-a08a1082545b-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:56:51.941308 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:51.941272 4951 generic.go:356] "Generic (PLEG): container finished" podID="da8a133c-4e96-4c80-aafa-a08a1082545b" containerID="8be600bc8e81c2bbadb72cec0556e43218493feeee46b688099ab831a0fe43fa" exitCode=0 Oct 30 07:56:51.941487 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:51.941327 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn" Oct 30 07:56:51.941487 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:51.941359 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn" event={"ID":"da8a133c-4e96-4c80-aafa-a08a1082545b","Type":"ContainerDied","Data":"8be600bc8e81c2bbadb72cec0556e43218493feeee46b688099ab831a0fe43fa"} Oct 30 07:56:51.941487 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:51.941422 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn" event={"ID":"da8a133c-4e96-4c80-aafa-a08a1082545b","Type":"ContainerDied","Data":"87041a89cc6da35620999557122f02df64f853cda1955d9bc14a5943bff88049"} Oct 30 07:56:51.941487 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:51.941453 4951 scope.go:122] "RemoveContainer" containerID="8be600bc8e81c2bbadb72cec0556e43218493feeee46b688099ab831a0fe43fa" Oct 30 07:56:51.952515 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:51.952495 4951 scope.go:122] "RemoveContainer" containerID="8be600bc8e81c2bbadb72cec0556e43218493feeee46b688099ab831a0fe43fa" Oct 30 07:56:51.952795 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:56:51.952774 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8be600bc8e81c2bbadb72cec0556e43218493feeee46b688099ab831a0fe43fa\": container with ID starting with 8be600bc8e81c2bbadb72cec0556e43218493feeee46b688099ab831a0fe43fa not found: ID does not exist" containerID="8be600bc8e81c2bbadb72cec0556e43218493feeee46b688099ab831a0fe43fa" Oct 30 07:56:51.952878 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:51.952803 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8be600bc8e81c2bbadb72cec0556e43218493feeee46b688099ab831a0fe43fa"} err="failed to get container status \"8be600bc8e81c2bbadb72cec0556e43218493feeee46b688099ab831a0fe43fa\": rpc error: code = NotFound desc = could not find container \"8be600bc8e81c2bbadb72cec0556e43218493feeee46b688099ab831a0fe43fa\": container with ID starting with 8be600bc8e81c2bbadb72cec0556e43218493feeee46b688099ab831a0fe43fa not found: ID does not exist" Oct 30 07:56:52.003003 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:52.002968 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn"] Oct 30 07:56:52.007448 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:52.007419 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fc3a23-85645c9bbd-pbqmn"] Oct 30 07:56:53.025170 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:56:53.025134 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:56:53.777750 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:56:53.777721 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da8a133c-4e96-4c80-aafa-a08a1082545b" path="/var/lib/kubelet/pods/da8a133c-4e96-4c80-aafa-a08a1082545b/volumes" Oct 30 07:57:23.027739 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:57:23.027651 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:57:51.311528 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:57:51.311495 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d"] Oct 30 07:57:51.312059 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:57:51.311778 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d" podUID="e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4" containerName="maas-api" containerID="cri-o://b9be4a40b792df9fa9e0dbc8fae95f6b1f0d95e6ca07aaad085a730a99ee7190" gracePeriod=30 Oct 30 07:57:51.721889 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:57:51.721671 4951 generic.go:356] "Generic (PLEG): container finished" podID="e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4" containerID="b9be4a40b792df9fa9e0dbc8fae95f6b1f0d95e6ca07aaad085a730a99ee7190" exitCode=0 Oct 30 07:57:51.721889 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:57:51.721735 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d" event={"ID":"e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4","Type":"ContainerDied","Data":"b9be4a40b792df9fa9e0dbc8fae95f6b1f0d95e6ca07aaad085a730a99ee7190"} Oct 30 07:57:51.760600 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:57:51.749162 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d" Oct 30 07:57:51.769465 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:57:51.769426 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4-metrics-certs\") pod \"e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4\" (UID: \"e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4\") " Oct 30 07:57:51.770261 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:57:51.770232 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4-kube-api-access-2hvc8\" (UniqueName: \"kubernetes.io/projected/e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4-kube-api-access-2hvc8\") pod \"e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4\" (UID: \"e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4\") " Oct 30 07:57:51.770752 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:57:51.770730 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4-openshift-service-ca\") pod \"e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4\" (UID: \"e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4\") " Oct 30 07:57:51.783722 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:57:51.783688 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4-maas-api-tls\") pod \"e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4\" (UID: \"e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4\") " Oct 30 07:57:51.784973 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:57:51.771159 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4-openshift-service-ca" pod "e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4" (UID: "e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:57:51.865675 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:57:51.836352 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4-metrics-certs" pod "e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4" (UID: "e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:57:51.889448 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:57:51.873368 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4-maas-api-tls" pod "e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4" (UID: "e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:57:51.906717 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:57:51.889847 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4-kube-api-access-2hvc8" pod "e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4" (UID: "e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4"). InnerVolumeSpecName "kube-api-access-2hvc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:57:51.906717 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:57:51.891444 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:57:51.906717 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:57:51.891475 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:57:51.906717 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:57:51.891489 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:57:51.906717 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:57:51.891503 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hvc8\" (UniqueName: \"kubernetes.io/projected/e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4-kube-api-access-2hvc8\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:57:52.737008 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:57:52.736926 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d" Oct 30 07:57:52.737008 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:57:52.736961 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d" event={"ID":"e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4","Type":"ContainerDied","Data":"52c6f5a1c742275bf4efc4486b46afbd97a1ab0b4633cdec5243f09dc18cccc0"} Oct 30 07:57:52.737560 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:57:52.737022 4951 scope.go:122] "RemoveContainer" containerID="b9be4a40b792df9fa9e0dbc8fae95f6b1f0d95e6ca07aaad085a730a99ee7190" Oct 30 07:57:52.795969 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:57:52.795938 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d"] Oct 30 07:57:52.809542 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:57:52.809518 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-759ae6-84cb8c8546-42f5d"] Oct 30 07:57:53.027317 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:57:53.027280 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:57:53.774261 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:57:53.774224 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4" path="/var/lib/kubelet/pods/e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4/volumes" Oct 30 07:57:59.393097 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:57:59.393057 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm"] Oct 30 07:57:59.393766 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:57:59.393288 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" podUID="afd14521-e339-4dcb-a824-9d8171138768" containerName="istio-proxy" containerID="cri-o://7501eddda9930e9bf0465095e72a76ad00cc2aadb5553635f5000d54784ba1ed" gracePeriod=30 Oct 30 07:57:59.408410 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:57:59.408365 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m"] Oct 30 07:57:59.408687 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:57:59.408646 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" podUID="16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37" containerName="istio-proxy" containerID="cri-o://c69d8059ca8406655ab15ea4adf03915fe2fc118aff65cd6c7ee76b89643a3d1" gracePeriod=30 Oct 30 07:58:01.100632 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:01.100592 4951 patch_prober.go:29] interesting pod/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:58:01.101156 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:01.100667 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" podUID="afd14521-e339-4dcb-a824-9d8171138768" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:58:04.812168 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:04.812136 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:58:04.857767 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:04.857729 4951 generic.go:356] "Generic (PLEG): container finished" podID="afd14521-e339-4dcb-a824-9d8171138768" containerID="7501eddda9930e9bf0465095e72a76ad00cc2aadb5553635f5000d54784ba1ed" exitCode=0 Oct 30 07:58:04.857966 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:04.857823 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" event={"ID":"afd14521-e339-4dcb-a824-9d8171138768","Type":"ContainerDied","Data":"7501eddda9930e9bf0465095e72a76ad00cc2aadb5553635f5000d54784ba1ed"} Oct 30 07:58:04.857966 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:04.857909 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" event={"ID":"afd14521-e339-4dcb-a824-9d8171138768","Type":"ContainerDied","Data":"ce8df9d8da833f98c2aaaa3fc9bd28b4407418a873c7d1eb716a56405d8fe2ae"} Oct 30 07:58:04.857966 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:04.857939 4951 scope.go:122] "RemoveContainer" containerID="7501eddda9930e9bf0465095e72a76ad00cc2aadb5553635f5000d54784ba1ed" Oct 30 07:58:04.858195 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:04.858113 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm" Oct 30 07:58:04.861096 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:04.861065 4951 generic.go:356] "Generic (PLEG): container finished" podID="16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37" containerID="c69d8059ca8406655ab15ea4adf03915fe2fc118aff65cd6c7ee76b89643a3d1" exitCode=0 Oct 30 07:58:04.861238 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:04.861130 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" event={"ID":"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37","Type":"ContainerDied","Data":"c69d8059ca8406655ab15ea4adf03915fe2fc118aff65cd6c7ee76b89643a3d1"} Oct 30 07:58:04.894418 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:04.894392 4951 scope.go:122] "RemoveContainer" containerID="c3b59c948fa7d3504e0440220a34f6d0171088fae364469414346869baf2f072" Oct 30 07:58:04.897219 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:04.897188 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:58:04.907252 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:04.907233 4951 scope.go:122] "RemoveContainer" containerID="7501eddda9930e9bf0465095e72a76ad00cc2aadb5553635f5000d54784ba1ed" Oct 30 07:58:04.907702 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:58:04.907680 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7501eddda9930e9bf0465095e72a76ad00cc2aadb5553635f5000d54784ba1ed\": container with ID starting with 7501eddda9930e9bf0465095e72a76ad00cc2aadb5553635f5000d54784ba1ed not found: ID does not exist" containerID="7501eddda9930e9bf0465095e72a76ad00cc2aadb5553635f5000d54784ba1ed" Oct 30 07:58:04.907772 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:04.907713 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7501eddda9930e9bf0465095e72a76ad00cc2aadb5553635f5000d54784ba1ed"} err="failed to get container status \"7501eddda9930e9bf0465095e72a76ad00cc2aadb5553635f5000d54784ba1ed\": rpc error: code = NotFound desc = could not find container \"7501eddda9930e9bf0465095e72a76ad00cc2aadb5553635f5000d54784ba1ed\": container with ID starting with 7501eddda9930e9bf0465095e72a76ad00cc2aadb5553635f5000d54784ba1ed not found: ID does not exist" Oct 30 07:58:04.907772 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:04.907733 4951 scope.go:122] "RemoveContainer" containerID="c3b59c948fa7d3504e0440220a34f6d0171088fae364469414346869baf2f072" Oct 30 07:58:04.908061 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:58:04.908043 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3b59c948fa7d3504e0440220a34f6d0171088fae364469414346869baf2f072\": container with ID starting with c3b59c948fa7d3504e0440220a34f6d0171088fae364469414346869baf2f072 not found: ID does not exist" containerID="c3b59c948fa7d3504e0440220a34f6d0171088fae364469414346869baf2f072" Oct 30 07:58:04.908140 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:04.908066 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3b59c948fa7d3504e0440220a34f6d0171088fae364469414346869baf2f072"} err="failed to get container status \"c3b59c948fa7d3504e0440220a34f6d0171088fae364469414346869baf2f072\": rpc error: code = NotFound desc = could not find container \"c3b59c948fa7d3504e0440220a34f6d0171088fae364469414346869baf2f072\": container with ID starting with c3b59c948fa7d3504e0440220a34f6d0171088fae364469414346869baf2f072 not found: ID does not exist" Oct 30 07:58:04.954542 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:04.954504 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-workload-socket\") pod \"afd14521-e339-4dcb-a824-9d8171138768\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " Oct 30 07:58:04.954772 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:04.954623 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afd14521-e339-4dcb-a824-9d8171138768-istio-token\" (UniqueName: \"kubernetes.io/projected/afd14521-e339-4dcb-a824-9d8171138768-istio-token\") pod \"afd14521-e339-4dcb-a824-9d8171138768\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " Oct 30 07:58:04.954772 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:04.954666 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-workload-certs\") pod \"afd14521-e339-4dcb-a824-9d8171138768\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " Oct 30 07:58:04.954772 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:04.954712 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/afd14521-e339-4dcb-a824-9d8171138768-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd14521-e339-4dcb-a824-9d8171138768-istio-podinfo\") pod \"afd14521-e339-4dcb-a824-9d8171138768\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " Oct 30 07:58:04.954960 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:04.954782 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-istio-envoy\") pod \"afd14521-e339-4dcb-a824-9d8171138768\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " Oct 30 07:58:04.954960 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:04.954824 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afd14521-e339-4dcb-a824-9d8171138768-kube-api-access-dtgq6\" (UniqueName: \"kubernetes.io/projected/afd14521-e339-4dcb-a824-9d8171138768-kube-api-access-dtgq6\") pod \"afd14521-e339-4dcb-a824-9d8171138768\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " Oct 30 07:58:04.954960 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:04.954863 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-istio-data\") pod \"afd14521-e339-4dcb-a824-9d8171138768\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " Oct 30 07:58:04.954960 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:04.954897 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-credential-socket\") pod \"afd14521-e339-4dcb-a824-9d8171138768\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " Oct 30 07:58:04.955191 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:04.954966 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afd14521-e339-4dcb-a824-9d8171138768-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd14521-e339-4dcb-a824-9d8171138768-istiod-ca-cert\") pod \"afd14521-e339-4dcb-a824-9d8171138768\" (UID: \"afd14521-e339-4dcb-a824-9d8171138768\") " Oct 30 07:58:04.956361 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:04.956115 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd14521-e339-4dcb-a824-9d8171138768-istiod-ca-cert" pod "afd14521-e339-4dcb-a824-9d8171138768" (UID: "afd14521-e339-4dcb-a824-9d8171138768"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:58:04.956361 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:04.956119 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-workload-socket" pod "afd14521-e339-4dcb-a824-9d8171138768" (UID: "afd14521-e339-4dcb-a824-9d8171138768"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:58:04.956804 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:04.956779 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-workload-certs" pod "afd14521-e339-4dcb-a824-9d8171138768" (UID: "afd14521-e339-4dcb-a824-9d8171138768"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:58:04.956894 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:04.956843 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-credential-socket" pod "afd14521-e339-4dcb-a824-9d8171138768" (UID: "afd14521-e339-4dcb-a824-9d8171138768"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:58:04.957155 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:04.957129 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-istio-data" pod "afd14521-e339-4dcb-a824-9d8171138768" (UID: "afd14521-e339-4dcb-a824-9d8171138768"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:58:04.958844 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:04.958802 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afd14521-e339-4dcb-a824-9d8171138768-istio-podinfo" pod "afd14521-e339-4dcb-a824-9d8171138768" (UID: "afd14521-e339-4dcb-a824-9d8171138768"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:58:04.959325 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:04.959293 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd14521-e339-4dcb-a824-9d8171138768-istio-token" pod "afd14521-e339-4dcb-a824-9d8171138768" (UID: "afd14521-e339-4dcb-a824-9d8171138768"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:58:04.959432 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:04.959409 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd14521-e339-4dcb-a824-9d8171138768-kube-api-access-dtgq6" pod "afd14521-e339-4dcb-a824-9d8171138768" (UID: "afd14521-e339-4dcb-a824-9d8171138768"). InnerVolumeSpecName "kube-api-access-dtgq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:58:04.959723 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:04.959702 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-istio-envoy" pod "afd14521-e339-4dcb-a824-9d8171138768" (UID: "afd14521-e339-4dcb-a824-9d8171138768"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:58:05.056869 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.056832 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istiod-ca-cert\") pod \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " Oct 30 07:58:05.057066 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.056884 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istio-envoy\") pod \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " Oct 30 07:58:05.057066 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.057005 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-kube-api-access-5nw5r\" (UniqueName: \"kubernetes.io/projected/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-kube-api-access-5nw5r\") pod \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " Oct 30 07:58:05.057175 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.057065 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-workload-certs\") pod \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " Oct 30 07:58:05.057242 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.057209 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-workload-socket\") pod \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " Oct 30 07:58:05.057378 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.057273 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istio-podinfo\") pod \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " Oct 30 07:58:05.057378 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.057273 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istiod-ca-cert" pod "16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37" (UID: "16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:58:05.057378 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.057346 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-credential-socket\") pod \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " Oct 30 07:58:05.057709 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.057432 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istio-token\" (UniqueName: \"kubernetes.io/projected/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istio-token\") pod \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " Oct 30 07:58:05.057821 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.057441 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-workload-certs" pod "16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37" (UID: "16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:58:05.057821 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.057560 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-workload-socket" pod "16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37" (UID: "16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:58:05.057821 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.057669 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-credential-socket" pod "16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37" (UID: "16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:58:05.058012 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.057908 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istio-data\") pod \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\" (UID: \"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37\") " Oct 30 07:58:05.058199 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.058166 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istio-data" pod "16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37" (UID: "16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:58:05.059719 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.059695 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:58:05.059814 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.059723 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd14521-e339-4dcb-a824-9d8171138768-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:58:05.059814 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.059736 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:58:05.059814 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.059749 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:58:05.059814 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.059763 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtgq6\" (UniqueName: \"kubernetes.io/projected/afd14521-e339-4dcb-a824-9d8171138768-kube-api-access-dtgq6\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:58:05.059814 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.059777 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:58:05.059814 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.059792 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:58:05.059814 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.059805 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd14521-e339-4dcb-a824-9d8171138768-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:58:05.059814 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.059802 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istio-token" pod "16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37" (UID: "16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:58:05.059814 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.059818 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:58:05.060174 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.059832 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:58:05.060174 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.059846 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:58:05.060174 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.059859 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd14521-e339-4dcb-a824-9d8171138768-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:58:05.060174 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.059874 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:58:05.060174 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.059887 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd14521-e339-4dcb-a824-9d8171138768-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:58:05.060174 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.059923 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-kube-api-access-5nw5r" pod "16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37" (UID: "16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37"). InnerVolumeSpecName "kube-api-access-5nw5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:58:05.060174 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.059974 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istio-podinfo" pod "16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37" (UID: "16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:58:05.060174 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.060098 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istio-envoy" pod "16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37" (UID: "16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:58:05.161542 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.161503 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:58:05.161542 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.161532 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:58:05.161542 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.161541 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:58:05.161542 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.161549 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nw5r\" (UniqueName: \"kubernetes.io/projected/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37-kube-api-access-5nw5r\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:58:05.210740 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.210708 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm"] Oct 30 07:58:05.217366 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.217338 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jv5rm"] Oct 30 07:58:05.775966 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.775934 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afd14521-e339-4dcb-a824-9d8171138768" path="/var/lib/kubelet/pods/afd14521-e339-4dcb-a824-9d8171138768/volumes" Oct 30 07:58:05.873785 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.873758 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" Oct 30 07:58:05.873785 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.873781 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m" event={"ID":"16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37","Type":"ContainerDied","Data":"7f045dbe2ad7ae31ea0d329682c775233b9bd7cf6948aab571c3d8c1043e1b03"} Oct 30 07:58:05.874293 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.873838 4951 scope.go:122] "RemoveContainer" containerID="c69d8059ca8406655ab15ea4adf03915fe2fc118aff65cd6c7ee76b89643a3d1" Oct 30 07:58:05.946529 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.946499 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m"] Oct 30 07:58:05.959537 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:05.959511 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-tvh4m"] Oct 30 07:58:07.775799 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:07.775767 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37" path="/var/lib/kubelet/pods/16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37/volumes" Oct 30 07:58:23.033285 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:58:23.033239 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:58:34.263375 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:34.263335 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47"] Oct 30 07:58:49.294501 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:49.294417 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47" podUID="d50edaa2-d402-4678-8d8b-e265ab2c65ed" containerName="main" containerID="cri-o://6d9760f9b4689a3ba7ce5642b0cf47bf6b03536cfee538509aec6fb0106462fe" gracePeriod=45 Oct 30 07:58:49.615321 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:49.615293 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47" Oct 30 07:58:49.654973 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:49.654938 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d50edaa2-d402-4678-8d8b-e265ab2c65ed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d50edaa2-d402-4678-8d8b-e265ab2c65ed-dshm\") pod \"d50edaa2-d402-4678-8d8b-e265ab2c65ed\" (UID: \"d50edaa2-d402-4678-8d8b-e265ab2c65ed\") " Oct 30 07:58:49.655155 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:49.655024 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d50edaa2-d402-4678-8d8b-e265ab2c65ed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d50edaa2-d402-4678-8d8b-e265ab2c65ed-tmp-dir\") pod \"d50edaa2-d402-4678-8d8b-e265ab2c65ed\" (UID: \"d50edaa2-d402-4678-8d8b-e265ab2c65ed\") " Oct 30 07:58:49.655155 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:49.655124 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d50edaa2-d402-4678-8d8b-e265ab2c65ed-kube-api-access-rvsrc\" (UniqueName: \"kubernetes.io/projected/d50edaa2-d402-4678-8d8b-e265ab2c65ed-kube-api-access-rvsrc\") pod \"d50edaa2-d402-4678-8d8b-e265ab2c65ed\" (UID: \"d50edaa2-d402-4678-8d8b-e265ab2c65ed\") " Oct 30 07:58:49.655299 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:49.655279 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d50edaa2-d402-4678-8d8b-e265ab2c65ed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d50edaa2-d402-4678-8d8b-e265ab2c65ed-model-cache\") pod \"d50edaa2-d402-4678-8d8b-e265ab2c65ed\" (UID: \"d50edaa2-d402-4678-8d8b-e265ab2c65ed\") " Oct 30 07:58:49.655299 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:49.655290 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50edaa2-d402-4678-8d8b-e265ab2c65ed-tmp-dir" pod "d50edaa2-d402-4678-8d8b-e265ab2c65ed" (UID: "d50edaa2-d402-4678-8d8b-e265ab2c65ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:58:49.655433 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:49.655413 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d50edaa2-d402-4678-8d8b-e265ab2c65ed-tls-certs\" (UniqueName: \"kubernetes.io/secret/d50edaa2-d402-4678-8d8b-e265ab2c65ed-tls-certs\") pod \"d50edaa2-d402-4678-8d8b-e265ab2c65ed\" (UID: \"d50edaa2-d402-4678-8d8b-e265ab2c65ed\") " Oct 30 07:58:49.655640 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:49.655452 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d50edaa2-d402-4678-8d8b-e265ab2c65ed-home\" (UniqueName: \"kubernetes.io/empty-dir/d50edaa2-d402-4678-8d8b-e265ab2c65ed-home\") pod \"d50edaa2-d402-4678-8d8b-e265ab2c65ed\" (UID: \"d50edaa2-d402-4678-8d8b-e265ab2c65ed\") " Oct 30 07:58:49.655713 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:49.655644 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50edaa2-d402-4678-8d8b-e265ab2c65ed-model-cache" pod "d50edaa2-d402-4678-8d8b-e265ab2c65ed" (UID: "d50edaa2-d402-4678-8d8b-e265ab2c65ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:58:49.655958 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:49.655902 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50edaa2-d402-4678-8d8b-e265ab2c65ed-home" pod "d50edaa2-d402-4678-8d8b-e265ab2c65ed" (UID: "d50edaa2-d402-4678-8d8b-e265ab2c65ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:58:49.656245 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:49.656224 4951 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d50edaa2-d402-4678-8d8b-e265ab2c65ed-model-cache\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:58:49.656338 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:49.656248 4951 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d50edaa2-d402-4678-8d8b-e265ab2c65ed-home\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:58:49.656338 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:49.656261 4951 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d50edaa2-d402-4678-8d8b-e265ab2c65ed-tmp-dir\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:58:49.657531 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:49.657491 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d50edaa2-d402-4678-8d8b-e265ab2c65ed-kube-api-access-rvsrc" pod "d50edaa2-d402-4678-8d8b-e265ab2c65ed" (UID: "d50edaa2-d402-4678-8d8b-e265ab2c65ed"). InnerVolumeSpecName "kube-api-access-rvsrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:58:49.657767 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:49.657739 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d50edaa2-d402-4678-8d8b-e265ab2c65ed-tls-certs" pod "d50edaa2-d402-4678-8d8b-e265ab2c65ed" (UID: "d50edaa2-d402-4678-8d8b-e265ab2c65ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:58:49.657849 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:49.657799 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50edaa2-d402-4678-8d8b-e265ab2c65ed-dshm" pod "d50edaa2-d402-4678-8d8b-e265ab2c65ed" (UID: "d50edaa2-d402-4678-8d8b-e265ab2c65ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:58:49.759265 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:49.758188 4951 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d50edaa2-d402-4678-8d8b-e265ab2c65ed-tls-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:58:49.759265 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:49.758225 4951 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d50edaa2-d402-4678-8d8b-e265ab2c65ed-dshm\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:58:49.759265 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:49.758239 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvsrc\" (UniqueName: \"kubernetes.io/projected/d50edaa2-d402-4678-8d8b-e265ab2c65ed-kube-api-access-rvsrc\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:58:49.856329 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:58:49.856241 4951 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd50edaa2_d402_4678_8d8b_e265ab2c65ed.slice/crio-0615c8116232bb282ac408c8452e097e263a647408a3d07e8687a9649e16b461\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd50edaa2_d402_4678_8d8b_e265ab2c65ed.slice\": RecentStats: unable to find data in memory cache]" Oct 30 07:58:50.276585 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:50.276541 4951 generic.go:356] "Generic (PLEG): container finished" podID="d50edaa2-d402-4678-8d8b-e265ab2c65ed" containerID="6d9760f9b4689a3ba7ce5642b0cf47bf6b03536cfee538509aec6fb0106462fe" exitCode=0 Oct 30 07:58:50.276798 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:50.276602 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47" event={"ID":"d50edaa2-d402-4678-8d8b-e265ab2c65ed","Type":"ContainerDied","Data":"6d9760f9b4689a3ba7ce5642b0cf47bf6b03536cfee538509aec6fb0106462fe"} Oct 30 07:58:50.276798 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:50.276635 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47" Oct 30 07:58:50.276798 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:50.276649 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47" event={"ID":"d50edaa2-d402-4678-8d8b-e265ab2c65ed","Type":"ContainerDied","Data":"0615c8116232bb282ac408c8452e097e263a647408a3d07e8687a9649e16b461"} Oct 30 07:58:50.276798 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:50.276670 4951 scope.go:122] "RemoveContainer" containerID="6d9760f9b4689a3ba7ce5642b0cf47bf6b03536cfee538509aec6fb0106462fe" Oct 30 07:58:50.289634 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:50.289602 4951 scope.go:122] "RemoveContainer" containerID="6d9760f9b4689a3ba7ce5642b0cf47bf6b03536cfee538509aec6fb0106462fe" Oct 30 07:58:50.290004 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:58:50.289973 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d9760f9b4689a3ba7ce5642b0cf47bf6b03536cfee538509aec6fb0106462fe\": container with ID starting with 6d9760f9b4689a3ba7ce5642b0cf47bf6b03536cfee538509aec6fb0106462fe not found: ID does not exist" containerID="6d9760f9b4689a3ba7ce5642b0cf47bf6b03536cfee538509aec6fb0106462fe" Oct 30 07:58:50.290111 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:50.290021 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d9760f9b4689a3ba7ce5642b0cf47bf6b03536cfee538509aec6fb0106462fe"} err="failed to get container status \"6d9760f9b4689a3ba7ce5642b0cf47bf6b03536cfee538509aec6fb0106462fe\": rpc error: code = NotFound desc = could not find container \"6d9760f9b4689a3ba7ce5642b0cf47bf6b03536cfee538509aec6fb0106462fe\": container with ID starting with 6d9760f9b4689a3ba7ce5642b0cf47bf6b03536cfee538509aec6fb0106462fe not found: ID does not exist" Oct 30 07:58:50.337620 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:50.337556 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47"] Oct 30 07:58:50.341266 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:50.341234 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-51147f/test-model-51147f-kserve-798f55f75-4gz47"] Oct 30 07:58:51.773891 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:51.773857 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d50edaa2-d402-4678-8d8b-e265ab2c65ed" path="/var/lib/kubelet/pods/d50edaa2-d402-4678-8d8b-e265ab2c65ed/volumes" Oct 30 07:58:52.536617 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:58:52.536506 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd"] Oct 30 07:58:53.025227 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:58:53.025189 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:59:07.564736 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:59:07.564685 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd" podUID="6f02b02a-47df-4c5b-a91d-36d9961e0676" containerName="main" containerID="cri-o://d61a68c952c08b4cd075408ada86dec965712d3d75776fffeb2b25e1786103aa" gracePeriod=45 Oct 30 07:59:07.872432 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:59:07.872401 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd" Oct 30 07:59:07.920112 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:59:07.920079 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f02b02a-47df-4c5b-a91d-36d9961e0676-tls-certs\" (UniqueName: \"kubernetes.io/secret/6f02b02a-47df-4c5b-a91d-36d9961e0676-tls-certs\") pod \"6f02b02a-47df-4c5b-a91d-36d9961e0676\" (UID: \"6f02b02a-47df-4c5b-a91d-36d9961e0676\") " Oct 30 07:59:07.920307 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:59:07.920139 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f02b02a-47df-4c5b-a91d-36d9961e0676-kube-api-access-tbwxx\" (UniqueName: \"kubernetes.io/projected/6f02b02a-47df-4c5b-a91d-36d9961e0676-kube-api-access-tbwxx\") pod \"6f02b02a-47df-4c5b-a91d-36d9961e0676\" (UID: \"6f02b02a-47df-4c5b-a91d-36d9961e0676\") " Oct 30 07:59:07.920307 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:59:07.920187 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f02b02a-47df-4c5b-a91d-36d9961e0676-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f02b02a-47df-4c5b-a91d-36d9961e0676-dshm\") pod \"6f02b02a-47df-4c5b-a91d-36d9961e0676\" (UID: \"6f02b02a-47df-4c5b-a91d-36d9961e0676\") " Oct 30 07:59:07.920307 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:59:07.920268 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f02b02a-47df-4c5b-a91d-36d9961e0676-home\" (UniqueName: \"kubernetes.io/empty-dir/6f02b02a-47df-4c5b-a91d-36d9961e0676-home\") pod \"6f02b02a-47df-4c5b-a91d-36d9961e0676\" (UID: \"6f02b02a-47df-4c5b-a91d-36d9961e0676\") " Oct 30 07:59:07.920307 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:59:07.920303 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f02b02a-47df-4c5b-a91d-36d9961e0676-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f02b02a-47df-4c5b-a91d-36d9961e0676-tmp-dir\") pod \"6f02b02a-47df-4c5b-a91d-36d9961e0676\" (UID: \"6f02b02a-47df-4c5b-a91d-36d9961e0676\") " Oct 30 07:59:07.920546 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:59:07.920375 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f02b02a-47df-4c5b-a91d-36d9961e0676-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f02b02a-47df-4c5b-a91d-36d9961e0676-model-cache\") pod \"6f02b02a-47df-4c5b-a91d-36d9961e0676\" (UID: \"6f02b02a-47df-4c5b-a91d-36d9961e0676\") " Oct 30 07:59:07.920876 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:59:07.920847 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f02b02a-47df-4c5b-a91d-36d9961e0676-home" pod "6f02b02a-47df-4c5b-a91d-36d9961e0676" (UID: "6f02b02a-47df-4c5b-a91d-36d9961e0676"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:59:07.921138 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:59:07.921080 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f02b02a-47df-4c5b-a91d-36d9961e0676-model-cache" pod "6f02b02a-47df-4c5b-a91d-36d9961e0676" (UID: "6f02b02a-47df-4c5b-a91d-36d9961e0676"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:59:07.921305 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:59:07.921285 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f02b02a-47df-4c5b-a91d-36d9961e0676-tmp-dir" pod "6f02b02a-47df-4c5b-a91d-36d9961e0676" (UID: "6f02b02a-47df-4c5b-a91d-36d9961e0676"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:59:07.921453 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:59:07.921436 4951 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f02b02a-47df-4c5b-a91d-36d9961e0676-home\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:59:07.921519 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:59:07.921457 4951 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f02b02a-47df-4c5b-a91d-36d9961e0676-tmp-dir\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:59:07.921519 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:59:07.921472 4951 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f02b02a-47df-4c5b-a91d-36d9961e0676-model-cache\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:59:07.928077 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:59:07.928045 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f02b02a-47df-4c5b-a91d-36d9961e0676-dshm" pod "6f02b02a-47df-4c5b-a91d-36d9961e0676" (UID: "6f02b02a-47df-4c5b-a91d-36d9961e0676"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:59:07.928241 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:59:07.928177 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f02b02a-47df-4c5b-a91d-36d9961e0676-kube-api-access-tbwxx" pod "6f02b02a-47df-4c5b-a91d-36d9961e0676" (UID: "6f02b02a-47df-4c5b-a91d-36d9961e0676"). InnerVolumeSpecName "kube-api-access-tbwxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:59:07.928885 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:59:07.928847 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f02b02a-47df-4c5b-a91d-36d9961e0676-tls-certs" pod "6f02b02a-47df-4c5b-a91d-36d9961e0676" (UID: "6f02b02a-47df-4c5b-a91d-36d9961e0676"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:59:08.023069 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:59:08.023037 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbwxx\" (UniqueName: \"kubernetes.io/projected/6f02b02a-47df-4c5b-a91d-36d9961e0676-kube-api-access-tbwxx\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:59:08.023069 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:59:08.023067 4951 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f02b02a-47df-4c5b-a91d-36d9961e0676-dshm\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:59:08.023282 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:59:08.023076 4951 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f02b02a-47df-4c5b-a91d-36d9961e0676-tls-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 07:59:08.504360 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:59:08.504328 4951 generic.go:356] "Generic (PLEG): container finished" podID="6f02b02a-47df-4c5b-a91d-36d9961e0676" containerID="d61a68c952c08b4cd075408ada86dec965712d3d75776fffeb2b25e1786103aa" exitCode=0 Oct 30 07:59:08.504614 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:59:08.504372 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd" Oct 30 07:59:08.504614 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:59:08.504411 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd" event={"ID":"6f02b02a-47df-4c5b-a91d-36d9961e0676","Type":"ContainerDied","Data":"d61a68c952c08b4cd075408ada86dec965712d3d75776fffeb2b25e1786103aa"} Oct 30 07:59:08.504614 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:59:08.504456 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd" event={"ID":"6f02b02a-47df-4c5b-a91d-36d9961e0676","Type":"ContainerDied","Data":"e4bc513e529a033ba4efae93a128615fae1caab6bb3356947ad353ffa83797b7"} Oct 30 07:59:08.504614 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:59:08.504476 4951 scope.go:122] "RemoveContainer" containerID="d61a68c952c08b4cd075408ada86dec965712d3d75776fffeb2b25e1786103aa" Oct 30 07:59:08.516511 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:59:08.516493 4951 scope.go:122] "RemoveContainer" containerID="d61a68c952c08b4cd075408ada86dec965712d3d75776fffeb2b25e1786103aa" Oct 30 07:59:08.516781 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:59:08.516763 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d61a68c952c08b4cd075408ada86dec965712d3d75776fffeb2b25e1786103aa\": container with ID starting with d61a68c952c08b4cd075408ada86dec965712d3d75776fffeb2b25e1786103aa not found: ID does not exist" containerID="d61a68c952c08b4cd075408ada86dec965712d3d75776fffeb2b25e1786103aa" Oct 30 07:59:08.516850 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:59:08.516789 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d61a68c952c08b4cd075408ada86dec965712d3d75776fffeb2b25e1786103aa"} err="failed to get container status \"d61a68c952c08b4cd075408ada86dec965712d3d75776fffeb2b25e1786103aa\": rpc error: code = NotFound desc = could not find container \"d61a68c952c08b4cd075408ada86dec965712d3d75776fffeb2b25e1786103aa\": container with ID starting with d61a68c952c08b4cd075408ada86dec965712d3d75776fffeb2b25e1786103aa not found: ID does not exist" Oct 30 07:59:08.566125 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:59:08.566084 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd"] Oct 30 07:59:08.607596 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:59:08.603621 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd"] Oct 30 07:59:08.760742 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:59:08.760646 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-268576/test-model-268576-kserve-78c6b97b75-jm5dd" podUID="6f02b02a-47df-4c5b-a91d-36d9961e0676" containerName="main" probeResult="failure" output="Get \"https://10.132.0.255:8000/ready\": context deadline exceeded" Oct 30 07:59:09.774519 ip-10-0-132-173 kubenswrapper[4951]: I1030 07:59:09.774485 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f02b02a-47df-4c5b-a91d-36d9961e0676" path="/var/lib/kubelet/pods/6f02b02a-47df-4c5b-a91d-36d9961e0676/volumes" Oct 30 07:59:23.026693 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:59:23.026640 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 07:59:53.024167 ip-10-0-132-173 kubenswrapper[4951]: E1030 07:59:53.024127 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 08:00:00.414370 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.414315 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff"] Oct 30 08:00:00.416496 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.416464 4951 cpu_manager.go:414] "containerMap: removing container" podUID="afd14521-e339-4dcb-a824-9d8171138768" containerName="istio-proxy" Oct 30 08:00:00.416708 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.416694 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afd14521-e339-4dcb-a824-9d8171138768" containerName="istio-proxy" Oct 30 08:00:00.416834 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.416822 4951 cpu_manager.go:414] "containerMap: removing container" podUID="6f02b02a-47df-4c5b-a91d-36d9961e0676" containerName="main" Oct 30 08:00:00.416919 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.416908 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f02b02a-47df-4c5b-a91d-36d9961e0676" containerName="main" Oct 30 08:00:00.417016 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.417007 4951 cpu_manager.go:414] "containerMap: removing container" podUID="e64255a8-c8c4-4ed3-b280-4d35314e746d" containerName="maas-api" Oct 30 08:00:00.417123 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.417112 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e64255a8-c8c4-4ed3-b280-4d35314e746d" containerName="maas-api" Oct 30 08:00:00.417231 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.417223 4951 cpu_manager.go:414] "containerMap: removing container" podUID="0e611ef5-131d-44f9-848d-3e9ae765f733" containerName="istio-proxy" Oct 30 08:00:00.417311 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.417302 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e611ef5-131d-44f9-848d-3e9ae765f733" containerName="istio-proxy" Oct 30 08:00:00.417396 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.417387 4951 cpu_manager.go:414] "containerMap: removing container" podUID="e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4" containerName="maas-api" Oct 30 08:00:00.417471 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.417462 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4" containerName="maas-api" Oct 30 08:00:00.417595 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.417582 4951 cpu_manager.go:414] "containerMap: removing container" podUID="da8a133c-4e96-4c80-aafa-a08a1082545b" containerName="maas-api" Oct 30 08:00:00.417683 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.417673 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da8a133c-4e96-4c80-aafa-a08a1082545b" containerName="maas-api" Oct 30 08:00:00.417769 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.417761 4951 cpu_manager.go:414] "containerMap: removing container" podUID="d50edaa2-d402-4678-8d8b-e265ab2c65ed" containerName="main" Oct 30 08:00:00.417844 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.417835 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d50edaa2-d402-4678-8d8b-e265ab2c65ed" containerName="main" Oct 30 08:00:00.417936 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.417927 4951 cpu_manager.go:414] "containerMap: removing container" podUID="16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37" containerName="istio-proxy" Oct 30 08:00:00.418016 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.418006 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37" containerName="istio-proxy" Oct 30 08:00:00.418099 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.418090 4951 cpu_manager.go:414] "containerMap: removing container" podUID="7771d5de-79de-4e6d-a0e8-acbacaf902af" containerName="istio-proxy" Oct 30 08:00:00.421264 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.421227 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7771d5de-79de-4e6d-a0e8-acbacaf902af" containerName="istio-proxy" Oct 30 08:00:00.421768 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.421747 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e611ef5-131d-44f9-848d-3e9ae765f733" containerName="istio-proxy" Oct 30 08:00:00.421919 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.421909 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f0998b-80d9-4e4f-a8ee-92e89e3a34a4" containerName="maas-api" Oct 30 08:00:00.422012 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.422002 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="da8a133c-4e96-4c80-aafa-a08a1082545b" containerName="maas-api" Oct 30 08:00:00.422090 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.422081 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="d50edaa2-d402-4678-8d8b-e265ab2c65ed" containerName="main" Oct 30 08:00:00.422168 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.422160 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="16d7ac8a-0bb5-44a8-8157-aa5ba2f8ec37" containerName="istio-proxy" Oct 30 08:00:00.422256 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.422247 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e611ef5-131d-44f9-848d-3e9ae765f733" containerName="istio-proxy" Oct 30 08:00:00.422340 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.422332 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="afd14521-e339-4dcb-a824-9d8171138768" containerName="istio-proxy" Oct 30 08:00:00.422416 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.422408 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="afd14521-e339-4dcb-a824-9d8171138768" containerName="istio-proxy" Oct 30 08:00:00.422498 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.422489 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="7771d5de-79de-4e6d-a0e8-acbacaf902af" containerName="istio-proxy" Oct 30 08:00:00.422598 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.422567 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f02b02a-47df-4c5b-a91d-36d9961e0676" containerName="main" Oct 30 08:00:00.422694 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.422684 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="e64255a8-c8c4-4ed3-b280-4d35314e746d" containerName="maas-api" Oct 30 08:00:00.428132 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.427981 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff" Oct 30 08:00:00.448760 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.448711 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:00:00.467949 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.467915 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff"] Oct 30 08:00:00.472643 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.472603 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m"] Oct 30 08:00:00.473821 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.473797 4951 cpu_manager.go:414] "containerMap: removing container" podUID="0e611ef5-131d-44f9-848d-3e9ae765f733" containerName="istio-proxy" Oct 30 08:00:00.473821 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.473820 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e611ef5-131d-44f9-848d-3e9ae765f733" containerName="istio-proxy" Oct 30 08:00:00.474025 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.473900 4951 cpu_manager.go:414] "containerMap: removing container" podUID="afd14521-e339-4dcb-a824-9d8171138768" containerName="istio-proxy" Oct 30 08:00:00.474025 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.473909 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afd14521-e339-4dcb-a824-9d8171138768" containerName="istio-proxy" Oct 30 08:00:00.479243 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.479186 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m" Oct 30 08:00:00.486690 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.486661 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs"] Oct 30 08:00:00.492529 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.492506 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs" Oct 30 08:00:00.506882 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.506857 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m"] Oct 30 08:00:00.512440 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.512406 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82220bd1-27c2-447d-967d-80032aa1480d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff\" (UID: \"82220bd1-27c2-447d-967d-80032aa1480d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff" Oct 30 08:00:00.512600 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.512493 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brk92\" (UniqueName: \"kubernetes.io/projected/d3ce0d73-117c-4309-9f1d-71a0a6c53299-kube-api-access-brk92\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs\" (UID: \"d3ce0d73-117c-4309-9f1d-71a0a6c53299\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs" Oct 30 08:00:00.512600 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.512558 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3933eee7-a582-4582-a7dc-23763490fad8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m\" (UID: \"3933eee7-a582-4582-a7dc-23763490fad8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m" Oct 30 08:00:00.512743 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.512609 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3ce0d73-117c-4309-9f1d-71a0a6c53299-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs\" (UID: \"d3ce0d73-117c-4309-9f1d-71a0a6c53299\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs" Oct 30 08:00:00.512743 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.512634 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c44bz\" (UniqueName: \"kubernetes.io/projected/3933eee7-a582-4582-a7dc-23763490fad8-kube-api-access-c44bz\") pod \"maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m\" (UID: \"3933eee7-a582-4582-a7dc-23763490fad8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m" Oct 30 08:00:00.512873 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.512856 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4zsm\" (UniqueName: \"kubernetes.io/projected/82220bd1-27c2-447d-967d-80032aa1480d-kube-api-access-c4zsm\") pod \"maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff\" (UID: \"82220bd1-27c2-447d-967d-80032aa1480d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff" Oct 30 08:00:00.519937 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.519911 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs"] Oct 30 08:00:00.614739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.614686 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4zsm\" (UniqueName: \"kubernetes.io/projected/82220bd1-27c2-447d-967d-80032aa1480d-kube-api-access-c4zsm\") pod \"maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff\" (UID: \"82220bd1-27c2-447d-967d-80032aa1480d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff" Oct 30 08:00:00.614966 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.614851 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82220bd1-27c2-447d-967d-80032aa1480d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff\" (UID: \"82220bd1-27c2-447d-967d-80032aa1480d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff" Oct 30 08:00:00.614966 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.614925 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brk92\" (UniqueName: \"kubernetes.io/projected/d3ce0d73-117c-4309-9f1d-71a0a6c53299-kube-api-access-brk92\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs\" (UID: \"d3ce0d73-117c-4309-9f1d-71a0a6c53299\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs" Oct 30 08:00:00.615103 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.614998 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3933eee7-a582-4582-a7dc-23763490fad8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m\" (UID: \"3933eee7-a582-4582-a7dc-23763490fad8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m" Oct 30 08:00:00.615103 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.615029 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3ce0d73-117c-4309-9f1d-71a0a6c53299-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs\" (UID: \"d3ce0d73-117c-4309-9f1d-71a0a6c53299\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs" Oct 30 08:00:00.615103 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.615056 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c44bz\" (UniqueName: \"kubernetes.io/projected/3933eee7-a582-4582-a7dc-23763490fad8-kube-api-access-c44bz\") pod \"maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m\" (UID: \"3933eee7-a582-4582-a7dc-23763490fad8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m" Oct 30 08:00:00.615690 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.615662 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82220bd1-27c2-447d-967d-80032aa1480d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff\" (UID: \"82220bd1-27c2-447d-967d-80032aa1480d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff" Oct 30 08:00:00.615846 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.615695 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3933eee7-a582-4582-a7dc-23763490fad8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m\" (UID: \"3933eee7-a582-4582-a7dc-23763490fad8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m" Oct 30 08:00:00.615846 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.615748 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3ce0d73-117c-4309-9f1d-71a0a6c53299-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs\" (UID: \"d3ce0d73-117c-4309-9f1d-71a0a6c53299\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs" Oct 30 08:00:00.645357 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.645314 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4zsm\" (UniqueName: \"kubernetes.io/projected/82220bd1-27c2-447d-967d-80032aa1480d-kube-api-access-c4zsm\") pod \"maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff\" (UID: \"82220bd1-27c2-447d-967d-80032aa1480d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff" Oct 30 08:00:00.645650 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.645628 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c44bz\" (UniqueName: \"kubernetes.io/projected/3933eee7-a582-4582-a7dc-23763490fad8-kube-api-access-c44bz\") pod \"maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m\" (UID: \"3933eee7-a582-4582-a7dc-23763490fad8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m" Oct 30 08:00:00.648313 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.648286 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brk92\" (UniqueName: \"kubernetes.io/projected/d3ce0d73-117c-4309-9f1d-71a0a6c53299-kube-api-access-brk92\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs\" (UID: \"d3ce0d73-117c-4309-9f1d-71a0a6c53299\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs" Oct 30 08:00:00.749636 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.749600 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff" Oct 30 08:00:00.801624 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.801583 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m" Oct 30 08:00:00.837843 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:00.837815 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs" Oct 30 08:00:01.067393 ip-10-0-132-173 kubenswrapper[4951]: W1030 08:00:01.067352 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82220bd1_27c2_447d_967d_80032aa1480d.slice/crio-7c897cf32363f1656a2243e76001a4d87697f747c605a770a917624d928c3d1b WatchSource:0}: Error finding container 7c897cf32363f1656a2243e76001a4d87697f747c605a770a917624d928c3d1b: Status 404 returned error can't find the container with id 7c897cf32363f1656a2243e76001a4d87697f747c605a770a917624d928c3d1b Oct 30 08:00:01.074834 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:01.074793 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff"] Oct 30 08:00:01.137682 ip-10-0-132-173 kubenswrapper[4951]: W1030 08:00:01.137655 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3933eee7_a582_4582_a7dc_23763490fad8.slice/crio-521190f6ca220e897e6369648f2d8705e0e36d3737f1363fdbb68d97315a7d82 WatchSource:0}: Error finding container 521190f6ca220e897e6369648f2d8705e0e36d3737f1363fdbb68d97315a7d82: Status 404 returned error can't find the container with id 521190f6ca220e897e6369648f2d8705e0e36d3737f1363fdbb68d97315a7d82 Oct 30 08:00:01.138497 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:01.138454 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m"] Oct 30 08:00:01.156034 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:01.155997 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m" event={"ID":"3933eee7-a582-4582-a7dc-23763490fad8","Type":"ContainerStarted","Data":"521190f6ca220e897e6369648f2d8705e0e36d3737f1363fdbb68d97315a7d82"} Oct 30 08:00:01.160129 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:01.158794 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff" event={"ID":"82220bd1-27c2-447d-967d-80032aa1480d","Type":"ContainerStarted","Data":"7c897cf32363f1656a2243e76001a4d87697f747c605a770a917624d928c3d1b"} Oct 30 08:00:01.167221 ip-10-0-132-173 kubenswrapper[4951]: W1030 08:00:01.166173 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ce0d73_117c_4309_9f1d_71a0a6c53299.slice/crio-1910dc701650ffa7a690ed3c96906212b245dca82c3b93229c98cc3ede511961 WatchSource:0}: Error finding container 1910dc701650ffa7a690ed3c96906212b245dca82c3b93229c98cc3ede511961: Status 404 returned error can't find the container with id 1910dc701650ffa7a690ed3c96906212b245dca82c3b93229c98cc3ede511961 Oct 30 08:00:01.172839 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:01.172814 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs"] Oct 30 08:00:02.169968 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:02.169932 4951 generic.go:356] "Generic (PLEG): container finished" podID="3933eee7-a582-4582-a7dc-23763490fad8" containerID="555a8b2e78973a42da927ef3adcf295fbf15f5b4a4e7cee84d7ce9692a489be7" exitCode=60 Oct 30 08:00:02.170540 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:02.169986 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m" event={"ID":"3933eee7-a582-4582-a7dc-23763490fad8","Type":"ContainerDied","Data":"555a8b2e78973a42da927ef3adcf295fbf15f5b4a4e7cee84d7ce9692a489be7"} Oct 30 08:00:02.170540 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:02.170407 4951 scope.go:122] "RemoveContainer" containerID="555a8b2e78973a42da927ef3adcf295fbf15f5b4a4e7cee84d7ce9692a489be7" Oct 30 08:00:02.172940 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:02.172915 4951 generic.go:356] "Generic (PLEG): container finished" podID="82220bd1-27c2-447d-967d-80032aa1480d" containerID="edf1ddde3417f01a612b461f5f2da58b0c43265672ef4c13b1dd1bed310db364" exitCode=60 Oct 30 08:00:02.173083 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:02.172987 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff" event={"ID":"82220bd1-27c2-447d-967d-80032aa1480d","Type":"ContainerDied","Data":"edf1ddde3417f01a612b461f5f2da58b0c43265672ef4c13b1dd1bed310db364"} Oct 30 08:00:02.173392 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:02.173373 4951 scope.go:122] "RemoveContainer" containerID="edf1ddde3417f01a612b461f5f2da58b0c43265672ef4c13b1dd1bed310db364" Oct 30 08:00:02.174979 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:02.174931 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs" event={"ID":"d3ce0d73-117c-4309-9f1d-71a0a6c53299","Type":"ContainerStarted","Data":"4f36f34b325263eda8074afbd40a501f89fa00c07597323e45785511e6e8920d"} Oct 30 08:00:02.175092 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:02.174989 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs" event={"ID":"d3ce0d73-117c-4309-9f1d-71a0a6c53299","Type":"ContainerStarted","Data":"1910dc701650ffa7a690ed3c96906212b245dca82c3b93229c98cc3ede511961"} Oct 30 08:00:02.266654 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:02.266606 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs" podStartSLOduration=2.266590484 podStartE2EDuration="2.266590484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:02.262641695 +0000 UTC m=+3808.605689661" watchObservedRunningTime="2026-10-30 08:00:02.266590484 +0000 UTC m=+3808.609638446" Oct 30 08:00:03.185073 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:03.184959 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff" event={"ID":"82220bd1-27c2-447d-967d-80032aa1480d","Type":"ContainerStarted","Data":"adaf9dde67f851781c0d698898d0550ea53ea3645cb34c3a621e4309ecd85b69"} Oct 30 08:00:03.186847 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:03.186818 4951 generic.go:356] "Generic (PLEG): container finished" podID="d3ce0d73-117c-4309-9f1d-71a0a6c53299" containerID="4f36f34b325263eda8074afbd40a501f89fa00c07597323e45785511e6e8920d" exitCode=60 Oct 30 08:00:03.186991 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:03.186902 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs" event={"ID":"d3ce0d73-117c-4309-9f1d-71a0a6c53299","Type":"ContainerDied","Data":"4f36f34b325263eda8074afbd40a501f89fa00c07597323e45785511e6e8920d"} Oct 30 08:00:03.187234 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:03.187211 4951 scope.go:122] "RemoveContainer" containerID="4f36f34b325263eda8074afbd40a501f89fa00c07597323e45785511e6e8920d" Oct 30 08:00:03.189368 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:03.189334 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m" event={"ID":"3933eee7-a582-4582-a7dc-23763490fad8","Type":"ContainerStarted","Data":"7f286d0cfee5724573d23e5c00b7223ae0c310f15e07788a5439e319bc3efc07"} Oct 30 08:00:03.236516 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:03.236459 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff" podStartSLOduration=3.236441272 podStartE2EDuration="3.236441272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:03.233745357 +0000 UTC m=+3809.576793324" watchObservedRunningTime="2026-10-30 08:00:03.236441272 +0000 UTC m=+3809.579489237" Oct 30 08:00:03.332354 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:03.332304 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m" podStartSLOduration=3.332289847 podStartE2EDuration="3.332289847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:03.327753924 +0000 UTC m=+3809.670801889" watchObservedRunningTime="2026-10-30 08:00:03.332289847 +0000 UTC m=+3809.675337813" Oct 30 08:00:04.199147 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:04.199107 4951 generic.go:356] "Generic (PLEG): container finished" podID="82220bd1-27c2-447d-967d-80032aa1480d" containerID="adaf9dde67f851781c0d698898d0550ea53ea3645cb34c3a621e4309ecd85b69" exitCode=60 Oct 30 08:00:04.199657 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:04.199156 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff" event={"ID":"82220bd1-27c2-447d-967d-80032aa1480d","Type":"ContainerDied","Data":"adaf9dde67f851781c0d698898d0550ea53ea3645cb34c3a621e4309ecd85b69"} Oct 30 08:00:04.199657 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:04.199229 4951 scope.go:122] "RemoveContainer" containerID="edf1ddde3417f01a612b461f5f2da58b0c43265672ef4c13b1dd1bed310db364" Oct 30 08:00:04.199794 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:04.199661 4951 scope.go:122] "RemoveContainer" containerID="adaf9dde67f851781c0d698898d0550ea53ea3645cb34c3a621e4309ecd85b69" Oct 30 08:00:04.200147 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:00:04.200107 4951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff_odh-ai-gateway-infra(82220bd1-27c2-447d-967d-80032aa1480d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff" podUID="82220bd1-27c2-447d-967d-80032aa1480d" Oct 30 08:00:04.201758 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:04.201730 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs" event={"ID":"d3ce0d73-117c-4309-9f1d-71a0a6c53299","Type":"ContainerStarted","Data":"44de972d01c56e0c9272715adf4b060be2874bf725e9e923d03acbe308543f58"} Oct 30 08:00:04.210049 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:04.210006 4951 generic.go:356] "Generic (PLEG): container finished" podID="3933eee7-a582-4582-a7dc-23763490fad8" containerID="7f286d0cfee5724573d23e5c00b7223ae0c310f15e07788a5439e319bc3efc07" exitCode=60 Oct 30 08:00:04.210267 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:04.210099 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m" event={"ID":"3933eee7-a582-4582-a7dc-23763490fad8","Type":"ContainerDied","Data":"7f286d0cfee5724573d23e5c00b7223ae0c310f15e07788a5439e319bc3efc07"} Oct 30 08:00:04.210494 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:04.210476 4951 scope.go:122] "RemoveContainer" containerID="7f286d0cfee5724573d23e5c00b7223ae0c310f15e07788a5439e319bc3efc07" Oct 30 08:00:04.210954 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:00:04.210930 4951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m_odh-ai-gateway-infra(3933eee7-a582-4582-a7dc-23763490fad8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m" podUID="3933eee7-a582-4582-a7dc-23763490fad8" Oct 30 08:00:04.219155 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:04.219131 4951 scope.go:122] "RemoveContainer" containerID="555a8b2e78973a42da927ef3adcf295fbf15f5b4a4e7cee84d7ce9692a489be7" Oct 30 08:00:05.220081 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:05.220049 4951 scope.go:122] "RemoveContainer" containerID="7f286d0cfee5724573d23e5c00b7223ae0c310f15e07788a5439e319bc3efc07" Oct 30 08:00:05.220559 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:00:05.220482 4951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m_odh-ai-gateway-infra(3933eee7-a582-4582-a7dc-23763490fad8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m" podUID="3933eee7-a582-4582-a7dc-23763490fad8" Oct 30 08:00:05.221858 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:05.221832 4951 scope.go:122] "RemoveContainer" containerID="adaf9dde67f851781c0d698898d0550ea53ea3645cb34c3a621e4309ecd85b69" Oct 30 08:00:05.222243 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:00:05.222217 4951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff_odh-ai-gateway-infra(82220bd1-27c2-447d-967d-80032aa1480d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff" podUID="82220bd1-27c2-447d-967d-80032aa1480d" Oct 30 08:00:05.223722 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:05.223690 4951 generic.go:356] "Generic (PLEG): container finished" podID="d3ce0d73-117c-4309-9f1d-71a0a6c53299" containerID="44de972d01c56e0c9272715adf4b060be2874bf725e9e923d03acbe308543f58" exitCode=60 Oct 30 08:00:05.223847 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:05.223768 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs" event={"ID":"d3ce0d73-117c-4309-9f1d-71a0a6c53299","Type":"ContainerDied","Data":"44de972d01c56e0c9272715adf4b060be2874bf725e9e923d03acbe308543f58"} Oct 30 08:00:05.223847 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:05.223827 4951 scope.go:122] "RemoveContainer" containerID="4f36f34b325263eda8074afbd40a501f89fa00c07597323e45785511e6e8920d" Oct 30 08:00:05.224088 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:05.224070 4951 scope.go:122] "RemoveContainer" containerID="44de972d01c56e0c9272715adf4b060be2874bf725e9e923d03acbe308543f58" Oct 30 08:00:05.224472 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:00:05.224449 4951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs_odh-ai-gateway-infra(d3ce0d73-117c-4309-9f1d-71a0a6c53299)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs" podUID="d3ce0d73-117c-4309-9f1d-71a0a6c53299" Oct 30 08:00:06.236162 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:06.236130 4951 scope.go:122] "RemoveContainer" containerID="44de972d01c56e0c9272715adf4b060be2874bf725e9e923d03acbe308543f58" Oct 30 08:00:06.236724 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:00:06.236513 4951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs_odh-ai-gateway-infra(d3ce0d73-117c-4309-9f1d-71a0a6c53299)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs" podUID="d3ce0d73-117c-4309-9f1d-71a0a6c53299" Oct 30 08:00:07.388062 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:07.388026 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-352fb3-589f6cb4d8-sdbxs"] Oct 30 08:00:07.426309 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:07.426279 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-352fb3-67f655f77f-xzf5w"] Oct 30 08:00:12.388878 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:12.388837 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-352fb3-589f6cb4d8-sdbxs" podUID="1ad0084a-1d54-4b23-ac5c-99cbc8b6da24" containerName="payload-processing" containerID="cri-o://31b368458fa1f8450997a7f6c62de962b68332528d496f8d974c1f0bf50aa995" gracePeriod=25 Oct 30 08:00:12.427522 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:12.427451 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-352fb3-67f655f77f-xzf5w" podUID="cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2" containerName="payload-pre-processing" containerID="cri-o://38d12521fb5c6c835cbc258c388583398a6b5fd114e8ac2ce3ebf1193e5bf67c" gracePeriod=25 Oct 30 08:00:12.739846 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:12.739810 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-352fb3-589f6cb4d8-sdbxs" Oct 30 08:00:12.855151 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:12.855108 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ad0084a-1d54-4b23-ac5c-99cbc8b6da24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad0084a-1d54-4b23-ac5c-99cbc8b6da24-plugins-config-volume\") pod \"1ad0084a-1d54-4b23-ac5c-99cbc8b6da24\" (UID: \"1ad0084a-1d54-4b23-ac5c-99cbc8b6da24\") " Oct 30 08:00:12.855356 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:12.855247 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ad0084a-1d54-4b23-ac5c-99cbc8b6da24-kube-api-access-q9mcc\" (UniqueName: \"kubernetes.io/projected/1ad0084a-1d54-4b23-ac5c-99cbc8b6da24-kube-api-access-q9mcc\") pod \"1ad0084a-1d54-4b23-ac5c-99cbc8b6da24\" (UID: \"1ad0084a-1d54-4b23-ac5c-99cbc8b6da24\") " Oct 30 08:00:12.855606 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:12.855563 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ad0084a-1d54-4b23-ac5c-99cbc8b6da24-plugins-config-volume" pod "1ad0084a-1d54-4b23-ac5c-99cbc8b6da24" (UID: "1ad0084a-1d54-4b23-ac5c-99cbc8b6da24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:12.856219 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:12.856193 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad0084a-1d54-4b23-ac5c-99cbc8b6da24-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:00:12.858204 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:12.858178 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ad0084a-1d54-4b23-ac5c-99cbc8b6da24-kube-api-access-q9mcc" pod "1ad0084a-1d54-4b23-ac5c-99cbc8b6da24" (UID: "1ad0084a-1d54-4b23-ac5c-99cbc8b6da24"). InnerVolumeSpecName "kube-api-access-q9mcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:12.957776 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:12.957692 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9mcc\" (UniqueName: \"kubernetes.io/projected/1ad0084a-1d54-4b23-ac5c-99cbc8b6da24-kube-api-access-q9mcc\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:00:13.329043 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:13.328998 4951 generic.go:356] "Generic (PLEG): container finished" podID="1ad0084a-1d54-4b23-ac5c-99cbc8b6da24" containerID="31b368458fa1f8450997a7f6c62de962b68332528d496f8d974c1f0bf50aa995" exitCode=0 Oct 30 08:00:13.329224 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:13.329056 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-352fb3-589f6cb4d8-sdbxs" Oct 30 08:00:13.329224 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:13.329074 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-352fb3-589f6cb4d8-sdbxs" event={"ID":"1ad0084a-1d54-4b23-ac5c-99cbc8b6da24","Type":"ContainerDied","Data":"31b368458fa1f8450997a7f6c62de962b68332528d496f8d974c1f0bf50aa995"} Oct 30 08:00:13.329224 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:13.329158 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-352fb3-589f6cb4d8-sdbxs" event={"ID":"1ad0084a-1d54-4b23-ac5c-99cbc8b6da24","Type":"ContainerDied","Data":"741df68671dba5549c2cfdbae6cd08d48c344eee4cde2c7496f18181db0069d7"} Oct 30 08:00:13.329224 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:13.329181 4951 scope.go:122] "RemoveContainer" containerID="31b368458fa1f8450997a7f6c62de962b68332528d496f8d974c1f0bf50aa995" Oct 30 08:00:13.343332 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:13.343309 4951 scope.go:122] "RemoveContainer" containerID="31b368458fa1f8450997a7f6c62de962b68332528d496f8d974c1f0bf50aa995" Oct 30 08:00:13.343780 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:00:13.343751 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31b368458fa1f8450997a7f6c62de962b68332528d496f8d974c1f0bf50aa995\": container with ID starting with 31b368458fa1f8450997a7f6c62de962b68332528d496f8d974c1f0bf50aa995 not found: ID does not exist" containerID="31b368458fa1f8450997a7f6c62de962b68332528d496f8d974c1f0bf50aa995" Oct 30 08:00:13.343925 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:13.343787 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31b368458fa1f8450997a7f6c62de962b68332528d496f8d974c1f0bf50aa995"} err="failed to get container status \"31b368458fa1f8450997a7f6c62de962b68332528d496f8d974c1f0bf50aa995\": rpc error: code = NotFound desc = could not find container \"31b368458fa1f8450997a7f6c62de962b68332528d496f8d974c1f0bf50aa995\": container with ID starting with 31b368458fa1f8450997a7f6c62de962b68332528d496f8d974c1f0bf50aa995 not found: ID does not exist" Oct 30 08:00:13.431759 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:13.431720 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-352fb3-589f6cb4d8-sdbxs"] Oct 30 08:00:13.450132 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:13.450098 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-352fb3-589f6cb4d8-sdbxs"] Oct 30 08:00:13.776535 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:13.776492 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ad0084a-1d54-4b23-ac5c-99cbc8b6da24" path="/var/lib/kubelet/pods/1ad0084a-1d54-4b23-ac5c-99cbc8b6da24/volumes" Oct 30 08:00:14.764074 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:14.764041 4951 scope.go:122] "RemoveContainer" containerID="adaf9dde67f851781c0d698898d0550ea53ea3645cb34c3a621e4309ecd85b69" Oct 30 08:00:15.350063 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:15.350025 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff" event={"ID":"82220bd1-27c2-447d-967d-80032aa1480d","Type":"ContainerStarted","Data":"b61f485a96e94a1016a66f9b3f806b217cc6bff1b683718182b80229a7cf2725"} Oct 30 08:00:15.394942 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:15.394836 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-352fb3-67f655f77f-xzf5w" podUID="cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:00:15.765104 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:15.765073 4951 scope.go:122] "RemoveContainer" containerID="44de972d01c56e0c9272715adf4b060be2874bf725e9e923d03acbe308543f58" Oct 30 08:00:16.361236 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:16.361154 4951 generic.go:356] "Generic (PLEG): container finished" podID="82220bd1-27c2-447d-967d-80032aa1480d" containerID="b61f485a96e94a1016a66f9b3f806b217cc6bff1b683718182b80229a7cf2725" exitCode=60 Oct 30 08:00:16.361398 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:16.361229 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff" event={"ID":"82220bd1-27c2-447d-967d-80032aa1480d","Type":"ContainerDied","Data":"b61f485a96e94a1016a66f9b3f806b217cc6bff1b683718182b80229a7cf2725"} Oct 30 08:00:16.361398 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:16.361293 4951 scope.go:122] "RemoveContainer" containerID="adaf9dde67f851781c0d698898d0550ea53ea3645cb34c3a621e4309ecd85b69" Oct 30 08:00:16.361758 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:16.361735 4951 scope.go:122] "RemoveContainer" containerID="b61f485a96e94a1016a66f9b3f806b217cc6bff1b683718182b80229a7cf2725" Oct 30 08:00:16.362179 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:00:16.362152 4951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff_odh-ai-gateway-infra(82220bd1-27c2-447d-967d-80032aa1480d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff" podUID="82220bd1-27c2-447d-967d-80032aa1480d" Oct 30 08:00:16.363937 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:16.363915 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs" event={"ID":"d3ce0d73-117c-4309-9f1d-71a0a6c53299","Type":"ContainerStarted","Data":"a428d332d24a89fa5edb5631448e9eac714412a3f417aed8972dce905ca875cc"} Oct 30 08:00:16.406105 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:16.406080 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff"] Oct 30 08:00:17.375769 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:17.375735 4951 generic.go:356] "Generic (PLEG): container finished" podID="d3ce0d73-117c-4309-9f1d-71a0a6c53299" containerID="a428d332d24a89fa5edb5631448e9eac714412a3f417aed8972dce905ca875cc" exitCode=60 Oct 30 08:00:17.376264 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:17.375784 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs" event={"ID":"d3ce0d73-117c-4309-9f1d-71a0a6c53299","Type":"ContainerDied","Data":"a428d332d24a89fa5edb5631448e9eac714412a3f417aed8972dce905ca875cc"} Oct 30 08:00:17.376264 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:17.375852 4951 scope.go:122] "RemoveContainer" containerID="44de972d01c56e0c9272715adf4b060be2874bf725e9e923d03acbe308543f58" Oct 30 08:00:17.376533 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:17.376269 4951 scope.go:122] "RemoveContainer" containerID="a428d332d24a89fa5edb5631448e9eac714412a3f417aed8972dce905ca875cc" Oct 30 08:00:17.376732 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:00:17.376689 4951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs_odh-ai-gateway-infra(d3ce0d73-117c-4309-9f1d-71a0a6c53299)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs" podUID="d3ce0d73-117c-4309-9f1d-71a0a6c53299" Oct 30 08:00:17.481150 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:17.481114 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs"] Oct 30 08:00:17.563511 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:17.563479 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff" Oct 30 08:00:17.634650 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:17.634551 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82220bd1-27c2-447d-967d-80032aa1480d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82220bd1-27c2-447d-967d-80032aa1480d-maas-api-service-ca\") pod \"82220bd1-27c2-447d-967d-80032aa1480d\" (UID: \"82220bd1-27c2-447d-967d-80032aa1480d\") " Oct 30 08:00:17.634875 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:17.634856 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82220bd1-27c2-447d-967d-80032aa1480d-kube-api-access-c4zsm\" (UniqueName: \"kubernetes.io/projected/82220bd1-27c2-447d-967d-80032aa1480d-kube-api-access-c4zsm\") pod \"82220bd1-27c2-447d-967d-80032aa1480d\" (UID: \"82220bd1-27c2-447d-967d-80032aa1480d\") " Oct 30 08:00:17.634995 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:17.634965 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82220bd1-27c2-447d-967d-80032aa1480d-maas-api-service-ca" pod "82220bd1-27c2-447d-967d-80032aa1480d" (UID: "82220bd1-27c2-447d-967d-80032aa1480d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:17.635470 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:17.635448 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82220bd1-27c2-447d-967d-80032aa1480d-maas-api-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:00:17.637870 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:17.637842 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82220bd1-27c2-447d-967d-80032aa1480d-kube-api-access-c4zsm" pod "82220bd1-27c2-447d-967d-80032aa1480d" (UID: "82220bd1-27c2-447d-967d-80032aa1480d"). InnerVolumeSpecName "kube-api-access-c4zsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:17.737542 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:17.737506 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4zsm\" (UniqueName: \"kubernetes.io/projected/82220bd1-27c2-447d-967d-80032aa1480d-kube-api-access-c4zsm\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:00:17.767016 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:17.764528 4951 scope.go:122] "RemoveContainer" containerID="7f286d0cfee5724573d23e5c00b7223ae0c310f15e07788a5439e319bc3efc07" Oct 30 08:00:18.385141 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:18.385105 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m" event={"ID":"3933eee7-a582-4582-a7dc-23763490fad8","Type":"ContainerStarted","Data":"4e1e85037d6d5a3957bc07857fb7495031c95b0a862a3ea3f8b1042b92c7f907"} Oct 30 08:00:18.386518 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:18.386498 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff" Oct 30 08:00:18.386518 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:18.386502 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff" event={"ID":"82220bd1-27c2-447d-967d-80032aa1480d","Type":"ContainerDied","Data":"7c897cf32363f1656a2243e76001a4d87697f747c605a770a917624d928c3d1b"} Oct 30 08:00:18.386719 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:18.386555 4951 scope.go:122] "RemoveContainer" containerID="b61f485a96e94a1016a66f9b3f806b217cc6bff1b683718182b80229a7cf2725" Oct 30 08:00:18.524094 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:18.524048 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff"] Oct 30 08:00:18.530528 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:18.530495 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889120-gslff"] Oct 30 08:00:18.579504 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:18.579477 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs" Oct 30 08:00:18.654453 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:18.654367 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3ce0d73-117c-4309-9f1d-71a0a6c53299-kube-api-access-brk92\" (UniqueName: \"kubernetes.io/projected/d3ce0d73-117c-4309-9f1d-71a0a6c53299-kube-api-access-brk92\") pod \"d3ce0d73-117c-4309-9f1d-71a0a6c53299\" (UID: \"d3ce0d73-117c-4309-9f1d-71a0a6c53299\") " Oct 30 08:00:18.654875 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:18.654847 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3ce0d73-117c-4309-9f1d-71a0a6c53299-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3ce0d73-117c-4309-9f1d-71a0a6c53299-maas-api-service-ca\") pod \"d3ce0d73-117c-4309-9f1d-71a0a6c53299\" (UID: \"d3ce0d73-117c-4309-9f1d-71a0a6c53299\") " Oct 30 08:00:18.655254 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:18.655227 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3ce0d73-117c-4309-9f1d-71a0a6c53299-maas-api-service-ca" pod "d3ce0d73-117c-4309-9f1d-71a0a6c53299" (UID: "d3ce0d73-117c-4309-9f1d-71a0a6c53299"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:18.655826 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:18.655803 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3ce0d73-117c-4309-9f1d-71a0a6c53299-maas-api-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:00:18.657146 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:18.657119 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ce0d73-117c-4309-9f1d-71a0a6c53299-kube-api-access-brk92" pod "d3ce0d73-117c-4309-9f1d-71a0a6c53299" (UID: "d3ce0d73-117c-4309-9f1d-71a0a6c53299"). InnerVolumeSpecName "kube-api-access-brk92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:18.757726 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:18.757700 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brk92\" (UniqueName: \"kubernetes.io/projected/d3ce0d73-117c-4309-9f1d-71a0a6c53299-kube-api-access-brk92\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:00:19.403145 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:19.403110 4951 generic.go:356] "Generic (PLEG): container finished" podID="3933eee7-a582-4582-a7dc-23763490fad8" containerID="4e1e85037d6d5a3957bc07857fb7495031c95b0a862a3ea3f8b1042b92c7f907" exitCode=60 Oct 30 08:00:19.403656 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:19.403195 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m" event={"ID":"3933eee7-a582-4582-a7dc-23763490fad8","Type":"ContainerDied","Data":"4e1e85037d6d5a3957bc07857fb7495031c95b0a862a3ea3f8b1042b92c7f907"} Oct 30 08:00:19.403656 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:19.403262 4951 scope.go:122] "RemoveContainer" containerID="7f286d0cfee5724573d23e5c00b7223ae0c310f15e07788a5439e319bc3efc07" Oct 30 08:00:19.403809 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:19.403729 4951 scope.go:122] "RemoveContainer" containerID="4e1e85037d6d5a3957bc07857fb7495031c95b0a862a3ea3f8b1042b92c7f907" Oct 30 08:00:19.404130 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:00:19.404105 4951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m_odh-ai-gateway-infra(3933eee7-a582-4582-a7dc-23763490fad8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m" podUID="3933eee7-a582-4582-a7dc-23763490fad8" Oct 30 08:00:19.406660 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:19.406638 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs" event={"ID":"d3ce0d73-117c-4309-9f1d-71a0a6c53299","Type":"ContainerDied","Data":"1910dc701650ffa7a690ed3c96906212b245dca82c3b93229c98cc3ede511961"} Oct 30 08:00:19.406786 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:19.406718 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs" Oct 30 08:00:19.421140 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:19.421123 4951 scope.go:122] "RemoveContainer" containerID="a428d332d24a89fa5edb5631448e9eac714412a3f417aed8972dce905ca875cc" Oct 30 08:00:19.445513 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:19.445483 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m"] Oct 30 08:00:19.504156 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:19.504117 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs"] Oct 30 08:00:19.510543 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:19.510504 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889120-tqzzs"] Oct 30 08:00:19.777186 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:19.777154 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82220bd1-27c2-447d-967d-80032aa1480d" path="/var/lib/kubelet/pods/82220bd1-27c2-447d-967d-80032aa1480d/volumes" Oct 30 08:00:19.777800 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:19.777773 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3ce0d73-117c-4309-9f1d-71a0a6c53299" path="/var/lib/kubelet/pods/d3ce0d73-117c-4309-9f1d-71a0a6c53299/volumes" Oct 30 08:00:20.643253 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:20.643227 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m" Oct 30 08:00:20.689062 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:20.688947 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3933eee7-a582-4582-a7dc-23763490fad8-kube-api-access-c44bz\" (UniqueName: \"kubernetes.io/projected/3933eee7-a582-4582-a7dc-23763490fad8-kube-api-access-c44bz\") pod \"3933eee7-a582-4582-a7dc-23763490fad8\" (UID: \"3933eee7-a582-4582-a7dc-23763490fad8\") " Oct 30 08:00:20.689397 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:20.689255 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3933eee7-a582-4582-a7dc-23763490fad8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3933eee7-a582-4582-a7dc-23763490fad8-maas-api-service-ca\") pod \"3933eee7-a582-4582-a7dc-23763490fad8\" (UID: \"3933eee7-a582-4582-a7dc-23763490fad8\") " Oct 30 08:00:20.690359 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:20.690319 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3933eee7-a582-4582-a7dc-23763490fad8-maas-api-service-ca" pod "3933eee7-a582-4582-a7dc-23763490fad8" (UID: "3933eee7-a582-4582-a7dc-23763490fad8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:20.695119 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:20.695089 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3933eee7-a582-4582-a7dc-23763490fad8-kube-api-access-c44bz" pod "3933eee7-a582-4582-a7dc-23763490fad8" (UID: "3933eee7-a582-4582-a7dc-23763490fad8"). InnerVolumeSpecName "kube-api-access-c44bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:20.792069 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:20.792033 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3933eee7-a582-4582-a7dc-23763490fad8-maas-api-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:00:20.792069 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:20.792068 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c44bz\" (UniqueName: \"kubernetes.io/projected/3933eee7-a582-4582-a7dc-23763490fad8-kube-api-access-c44bz\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:00:21.442614 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:21.442553 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m" event={"ID":"3933eee7-a582-4582-a7dc-23763490fad8","Type":"ContainerDied","Data":"521190f6ca220e897e6369648f2d8705e0e36d3737f1363fdbb68d97315a7d82"} Oct 30 08:00:21.442849 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:21.442625 4951 scope.go:122] "RemoveContainer" containerID="4e1e85037d6d5a3957bc07857fb7495031c95b0a862a3ea3f8b1042b92c7f907" Oct 30 08:00:21.442849 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:21.442559 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m" Oct 30 08:00:21.509592 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:21.509534 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m"] Oct 30 08:00:21.523179 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:21.523145 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889120-plx8m"] Oct 30 08:00:21.776185 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:21.776149 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3933eee7-a582-4582-a7dc-23763490fad8" path="/var/lib/kubelet/pods/3933eee7-a582-4582-a7dc-23763490fad8/volumes" Oct 30 08:00:23.024981 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:00:23.024949 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 08:00:25.395029 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:25.394994 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-352fb3-67f655f77f-xzf5w" podUID="cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:00:32.554056 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:32.554021 4951 generic.go:356] "Generic (PLEG): container finished" podID="cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2" containerID="38d12521fb5c6c835cbc258c388583398a6b5fd114e8ac2ce3ebf1193e5bf67c" exitCode=0 Oct 30 08:00:32.554454 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:32.554087 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-352fb3-67f655f77f-xzf5w" event={"ID":"cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2","Type":"ContainerDied","Data":"38d12521fb5c6c835cbc258c388583398a6b5fd114e8ac2ce3ebf1193e5bf67c"} Oct 30 08:00:32.735644 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:32.735614 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-352fb3-67f655f77f-xzf5w" Oct 30 08:00:32.837166 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:32.837127 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2-plugins-config-volume\") pod \"cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2\" (UID: \"cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2\") " Oct 30 08:00:32.837648 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:32.837538 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2-plugins-config-volume" pod "cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2" (UID: "cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:32.837948 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:32.837913 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2-kube-api-access-5jk29\" (UniqueName: \"kubernetes.io/projected/cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2-kube-api-access-5jk29\") pod \"cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2\" (UID: \"cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2\") " Oct 30 08:00:32.839131 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:32.839105 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:00:32.840460 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:32.840432 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2-kube-api-access-5jk29" pod "cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2" (UID: "cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2"). InnerVolumeSpecName "kube-api-access-5jk29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:32.941168 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:32.941087 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jk29\" (UniqueName: \"kubernetes.io/projected/cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2-kube-api-access-5jk29\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:00:33.564655 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:33.564621 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-352fb3-67f655f77f-xzf5w" event={"ID":"cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2","Type":"ContainerDied","Data":"65cc4373555bea2d0cca122a1890c5993392d12e0399d7dab1dcb4a4112b2bda"} Oct 30 08:00:33.564655 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:33.564649 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-352fb3-67f655f77f-xzf5w" Oct 30 08:00:33.565223 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:33.564686 4951 scope.go:122] "RemoveContainer" containerID="38d12521fb5c6c835cbc258c388583398a6b5fd114e8ac2ce3ebf1193e5bf67c" Oct 30 08:00:33.623629 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:33.623598 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-352fb3-67f655f77f-xzf5w"] Oct 30 08:00:33.637818 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:33.637793 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-352fb3-67f655f77f-xzf5w"] Oct 30 08:00:33.774426 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:33.774388 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2" path="/var/lib/kubelet/pods/cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2/volumes" Oct 30 08:00:53.025956 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:00:53.025921 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 08:00:54.712683 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:54.712644 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv"] Oct 30 08:00:54.713265 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:54.712871 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" podUID="90e2dc87-6736-4b39-acfe-8db6ee096cd5" containerName="istio-proxy" containerID="cri-o://be9f6e6fb74200d6194062f302eb451a724c724bdc2066aa5cfa5e0f1cb3e057" gracePeriod=30 Oct 30 08:00:55.704655 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:55.704618 4951 patch_prober.go:29] interesting pod/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 08:00:55.704838 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:00:55.704689 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" podUID="90e2dc87-6736-4b39-acfe-8db6ee096cd5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 08:01:00.024266 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.024220 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 08:01:00.048383 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.048353 4951 generic.go:356] "Generic (PLEG): container finished" podID="90e2dc87-6736-4b39-acfe-8db6ee096cd5" containerID="be9f6e6fb74200d6194062f302eb451a724c724bdc2066aa5cfa5e0f1cb3e057" exitCode=0 Oct 30 08:01:00.048638 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.048410 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" Oct 30 08:01:00.048638 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.048438 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" event={"ID":"90e2dc87-6736-4b39-acfe-8db6ee096cd5","Type":"ContainerDied","Data":"be9f6e6fb74200d6194062f302eb451a724c724bdc2066aa5cfa5e0f1cb3e057"} Oct 30 08:01:00.048638 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.048502 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv" event={"ID":"90e2dc87-6736-4b39-acfe-8db6ee096cd5","Type":"ContainerDied","Data":"31e7878602784b5b4d56b24b3d991a46fca3f119bfb16ea67d8f6305198c71fa"} Oct 30 08:01:00.048638 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.048531 4951 scope.go:122] "RemoveContainer" containerID="be9f6e6fb74200d6194062f302eb451a724c724bdc2066aa5cfa5e0f1cb3e057" Oct 30 08:01:00.060132 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.060099 4951 scope.go:122] "RemoveContainer" containerID="be9f6e6fb74200d6194062f302eb451a724c724bdc2066aa5cfa5e0f1cb3e057" Oct 30 08:01:00.060492 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:01:00.060470 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be9f6e6fb74200d6194062f302eb451a724c724bdc2066aa5cfa5e0f1cb3e057\": container with ID starting with be9f6e6fb74200d6194062f302eb451a724c724bdc2066aa5cfa5e0f1cb3e057 not found: ID does not exist" containerID="be9f6e6fb74200d6194062f302eb451a724c724bdc2066aa5cfa5e0f1cb3e057" Oct 30 08:01:00.060618 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.060505 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be9f6e6fb74200d6194062f302eb451a724c724bdc2066aa5cfa5e0f1cb3e057"} err="failed to get container status \"be9f6e6fb74200d6194062f302eb451a724c724bdc2066aa5cfa5e0f1cb3e057\": rpc error: code = NotFound desc = could not find container \"be9f6e6fb74200d6194062f302eb451a724c724bdc2066aa5cfa5e0f1cb3e057\": container with ID starting with be9f6e6fb74200d6194062f302eb451a724c724bdc2066aa5cfa5e0f1cb3e057 not found: ID does not exist" Oct 30 08:01:00.129271 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.129214 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istio-podinfo\") pod \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " Oct 30 08:01:00.129464 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.129289 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istiod-ca-cert\") pod \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " Oct 30 08:01:00.129464 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.129359 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istio-data\") pod \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " Oct 30 08:01:00.129617 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.129471 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-credential-socket\") pod \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " Oct 30 08:01:00.129688 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.129619 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istio-token\" (UniqueName: \"kubernetes.io/projected/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istio-token\") pod \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " Oct 30 08:01:00.129688 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.129678 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90e2dc87-6736-4b39-acfe-8db6ee096cd5-kube-api-access-2g9rh\" (UniqueName: \"kubernetes.io/projected/90e2dc87-6736-4b39-acfe-8db6ee096cd5-kube-api-access-2g9rh\") pod \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " Oct 30 08:01:00.129826 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.129807 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-workload-certs\") pod \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " Oct 30 08:01:00.130055 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.130033 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-workload-socket\") pod \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " Oct 30 08:01:00.130160 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.130109 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istio-envoy\") pod \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\" (UID: \"90e2dc87-6736-4b39-acfe-8db6ee096cd5\") " Oct 30 08:01:00.130443 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.130415 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-credential-socket" pod "90e2dc87-6736-4b39-acfe-8db6ee096cd5" (UID: "90e2dc87-6736-4b39-acfe-8db6ee096cd5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:01:00.130776 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.130750 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istiod-ca-cert" pod "90e2dc87-6736-4b39-acfe-8db6ee096cd5" (UID: "90e2dc87-6736-4b39-acfe-8db6ee096cd5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:01:00.130953 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.130930 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-workload-certs" pod "90e2dc87-6736-4b39-acfe-8db6ee096cd5" (UID: "90e2dc87-6736-4b39-acfe-8db6ee096cd5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:01:00.131051 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.131029 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-workload-socket" pod "90e2dc87-6736-4b39-acfe-8db6ee096cd5" (UID: "90e2dc87-6736-4b39-acfe-8db6ee096cd5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:01:00.131444 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.131417 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istio-data" pod "90e2dc87-6736-4b39-acfe-8db6ee096cd5" (UID: "90e2dc87-6736-4b39-acfe-8db6ee096cd5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:01:00.133174 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.133150 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:01:00.133275 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.133178 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:01:00.133275 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.133190 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:01:00.133275 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.133203 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:01:00.133275 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.133214 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:01:00.139097 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.139072 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istio-podinfo" pod "90e2dc87-6736-4b39-acfe-8db6ee096cd5" (UID: "90e2dc87-6736-4b39-acfe-8db6ee096cd5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:01:00.140709 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.140498 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istio-envoy" pod "90e2dc87-6736-4b39-acfe-8db6ee096cd5" (UID: "90e2dc87-6736-4b39-acfe-8db6ee096cd5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:01:00.140709 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.140684 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istio-token" pod "90e2dc87-6736-4b39-acfe-8db6ee096cd5" (UID: "90e2dc87-6736-4b39-acfe-8db6ee096cd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:01:00.141055 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.141021 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90e2dc87-6736-4b39-acfe-8db6ee096cd5-kube-api-access-2g9rh" pod "90e2dc87-6736-4b39-acfe-8db6ee096cd5" (UID: "90e2dc87-6736-4b39-acfe-8db6ee096cd5"). InnerVolumeSpecName "kube-api-access-2g9rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:01:00.234880 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.234782 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:01:00.234880 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.234828 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:01:00.234880 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.234844 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g9rh\" (UniqueName: \"kubernetes.io/projected/90e2dc87-6736-4b39-acfe-8db6ee096cd5-kube-api-access-2g9rh\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:01:00.234880 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.234862 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e2dc87-6736-4b39-acfe-8db6ee096cd5-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:01:00.406921 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.406888 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv"] Oct 30 08:01:00.410418 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:00.410396 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-sczfv"] Oct 30 08:01:01.776960 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:01.776927 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90e2dc87-6736-4b39-acfe-8db6ee096cd5" path="/var/lib/kubelet/pods/90e2dc87-6736-4b39-acfe-8db6ee096cd5/volumes" Oct 30 08:01:23.027157 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:01:23.027104 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 08:01:24.272600 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:24.268757 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t"] Oct 30 08:01:24.273133 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:24.272686 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t" podUID="abf16ce5-dabd-44dd-a441-9b7059c28338" containerName="maas-api" containerID="cri-o://62d745432c0b6accb3395712c7b5ef79480a64a5a87d0847b602c591c8e640ab" gracePeriod=30 Oct 30 08:01:24.907118 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:24.907089 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t" Oct 30 08:01:24.953305 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:24.952823 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abf16ce5-dabd-44dd-a441-9b7059c28338-kube-api-access-gfpnq\" (UniqueName: \"kubernetes.io/projected/abf16ce5-dabd-44dd-a441-9b7059c28338-kube-api-access-gfpnq\") pod \"abf16ce5-dabd-44dd-a441-9b7059c28338\" (UID: \"abf16ce5-dabd-44dd-a441-9b7059c28338\") " Oct 30 08:01:24.953509 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:24.953309 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abf16ce5-dabd-44dd-a441-9b7059c28338-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abf16ce5-dabd-44dd-a441-9b7059c28338-maas-api-tls\") pod \"abf16ce5-dabd-44dd-a441-9b7059c28338\" (UID: \"abf16ce5-dabd-44dd-a441-9b7059c28338\") " Oct 30 08:01:24.953509 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:24.953436 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abf16ce5-dabd-44dd-a441-9b7059c28338-metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf16ce5-dabd-44dd-a441-9b7059c28338-metrics-certs\") pod \"abf16ce5-dabd-44dd-a441-9b7059c28338\" (UID: \"abf16ce5-dabd-44dd-a441-9b7059c28338\") " Oct 30 08:01:24.953509 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:24.953484 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abf16ce5-dabd-44dd-a441-9b7059c28338-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf16ce5-dabd-44dd-a441-9b7059c28338-openshift-service-ca\") pod \"abf16ce5-dabd-44dd-a441-9b7059c28338\" (UID: \"abf16ce5-dabd-44dd-a441-9b7059c28338\") " Oct 30 08:01:24.973323 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:24.954699 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abf16ce5-dabd-44dd-a441-9b7059c28338-openshift-service-ca" pod "abf16ce5-dabd-44dd-a441-9b7059c28338" (UID: "abf16ce5-dabd-44dd-a441-9b7059c28338"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:01:24.973323 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:24.969841 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abf16ce5-dabd-44dd-a441-9b7059c28338-maas-api-tls" pod "abf16ce5-dabd-44dd-a441-9b7059c28338" (UID: "abf16ce5-dabd-44dd-a441-9b7059c28338"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:01:24.973906 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:24.973850 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abf16ce5-dabd-44dd-a441-9b7059c28338-metrics-certs" pod "abf16ce5-dabd-44dd-a441-9b7059c28338" (UID: "abf16ce5-dabd-44dd-a441-9b7059c28338"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:01:24.975314 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:24.975149 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf16ce5-dabd-44dd-a441-9b7059c28338-kube-api-access-gfpnq" pod "abf16ce5-dabd-44dd-a441-9b7059c28338" (UID: "abf16ce5-dabd-44dd-a441-9b7059c28338"). InnerVolumeSpecName "kube-api-access-gfpnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:01:25.055568 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:25.055530 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf16ce5-dabd-44dd-a441-9b7059c28338-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:01:25.055568 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:25.055568 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf16ce5-dabd-44dd-a441-9b7059c28338-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:01:25.055834 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:25.055599 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfpnq\" (UniqueName: \"kubernetes.io/projected/abf16ce5-dabd-44dd-a441-9b7059c28338-kube-api-access-gfpnq\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:01:25.055834 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:25.055613 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abf16ce5-dabd-44dd-a441-9b7059c28338-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:01:25.329984 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:25.329948 4951 generic.go:356] "Generic (PLEG): container finished" podID="abf16ce5-dabd-44dd-a441-9b7059c28338" containerID="62d745432c0b6accb3395712c7b5ef79480a64a5a87d0847b602c591c8e640ab" exitCode=0 Oct 30 08:01:25.330503 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:25.330025 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t" Oct 30 08:01:25.330503 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:25.330035 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t" event={"ID":"abf16ce5-dabd-44dd-a441-9b7059c28338","Type":"ContainerDied","Data":"62d745432c0b6accb3395712c7b5ef79480a64a5a87d0847b602c591c8e640ab"} Oct 30 08:01:25.330503 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:25.330135 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t" event={"ID":"abf16ce5-dabd-44dd-a441-9b7059c28338","Type":"ContainerDied","Data":"9f2c5660aead905de68650a37ce66b3e8cf973615d259291d7cf887bbd5f91ea"} Oct 30 08:01:25.330503 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:25.330171 4951 scope.go:122] "RemoveContainer" containerID="62d745432c0b6accb3395712c7b5ef79480a64a5a87d0847b602c591c8e640ab" Oct 30 08:01:25.343120 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:25.343088 4951 scope.go:122] "RemoveContainer" containerID="62d745432c0b6accb3395712c7b5ef79480a64a5a87d0847b602c591c8e640ab" Oct 30 08:01:25.343423 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:01:25.343402 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62d745432c0b6accb3395712c7b5ef79480a64a5a87d0847b602c591c8e640ab\": container with ID starting with 62d745432c0b6accb3395712c7b5ef79480a64a5a87d0847b602c591c8e640ab not found: ID does not exist" containerID="62d745432c0b6accb3395712c7b5ef79480a64a5a87d0847b602c591c8e640ab" Oct 30 08:01:25.343488 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:25.343436 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62d745432c0b6accb3395712c7b5ef79480a64a5a87d0847b602c591c8e640ab"} err="failed to get container status \"62d745432c0b6accb3395712c7b5ef79480a64a5a87d0847b602c591c8e640ab\": rpc error: code = NotFound desc = could not find container \"62d745432c0b6accb3395712c7b5ef79480a64a5a87d0847b602c591c8e640ab\": container with ID starting with 62d745432c0b6accb3395712c7b5ef79480a64a5a87d0847b602c591c8e640ab not found: ID does not exist" Oct 30 08:01:25.383053 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:25.383027 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t"] Oct 30 08:01:25.394108 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:25.394084 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a91a83-5b56c5c5c7-6b58t"] Oct 30 08:01:25.773717 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:25.773685 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abf16ce5-dabd-44dd-a441-9b7059c28338" path="/var/lib/kubelet/pods/abf16ce5-dabd-44dd-a441-9b7059c28338/volumes" Oct 30 08:01:34.739234 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:34.739197 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/3.log" Oct 30 08:01:34.745406 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:34.745378 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 08:01:34.753534 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:34.753513 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/3.log" Oct 30 08:01:34.758812 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:34.758790 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 08:01:34.813824 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:34.813784 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p"] Oct 30 08:01:34.814052 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:34.814028 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" podUID="d0c9697a-7dea-4662-8da4-c2618945d160" containerName="istio-proxy" containerID="cri-o://96f82bd1c8cc328ddf780900ba3326e9bf8394f6ae1ad18a83f832f3b8238e90" gracePeriod=30 Oct 30 08:01:40.127119 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.127089 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 08:01:40.239549 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.239499 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-workload-certs\") pod \"d0c9697a-7dea-4662-8da4-c2618945d160\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " Oct 30 08:01:40.239771 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.239561 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-istio-envoy\") pod \"d0c9697a-7dea-4662-8da4-c2618945d160\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " Oct 30 08:01:40.239851 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.239804 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-istio-data\") pod \"d0c9697a-7dea-4662-8da4-c2618945d160\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " Oct 30 08:01:40.239911 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.239832 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-workload-certs" pod "d0c9697a-7dea-4662-8da4-c2618945d160" (UID: "d0c9697a-7dea-4662-8da4-c2618945d160"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:01:40.239974 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.239915 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c9697a-7dea-4662-8da4-c2618945d160-kube-api-access-ffg47\" (UniqueName: \"kubernetes.io/projected/d0c9697a-7dea-4662-8da4-c2618945d160-kube-api-access-ffg47\") pod \"d0c9697a-7dea-4662-8da4-c2618945d160\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " Oct 30 08:01:40.239974 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.239954 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0c9697a-7dea-4662-8da4-c2618945d160-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c9697a-7dea-4662-8da4-c2618945d160-istio-podinfo\") pod \"d0c9697a-7dea-4662-8da4-c2618945d160\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " Oct 30 08:01:40.240101 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.240019 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-credential-socket\") pod \"d0c9697a-7dea-4662-8da4-c2618945d160\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " Oct 30 08:01:40.240101 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.240087 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c9697a-7dea-4662-8da4-c2618945d160-istio-token\" (UniqueName: \"kubernetes.io/projected/d0c9697a-7dea-4662-8da4-c2618945d160-istio-token\") pod \"d0c9697a-7dea-4662-8da4-c2618945d160\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " Oct 30 08:01:40.240219 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.240129 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-workload-socket\") pod \"d0c9697a-7dea-4662-8da4-c2618945d160\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " Oct 30 08:01:40.240219 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.240201 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0c9697a-7dea-4662-8da4-c2618945d160-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c9697a-7dea-4662-8da4-c2618945d160-istiod-ca-cert\") pod \"d0c9697a-7dea-4662-8da4-c2618945d160\" (UID: \"d0c9697a-7dea-4662-8da4-c2618945d160\") " Oct 30 08:01:40.240914 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.240886 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:01:40.241659 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.241636 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c9697a-7dea-4662-8da4-c2618945d160-istiod-ca-cert" pod "d0c9697a-7dea-4662-8da4-c2618945d160" (UID: "d0c9697a-7dea-4662-8da4-c2618945d160"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:01:40.242149 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.242122 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-credential-socket" pod "d0c9697a-7dea-4662-8da4-c2618945d160" (UID: "d0c9697a-7dea-4662-8da4-c2618945d160"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:01:40.242260 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.241876 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-istio-data" pod "d0c9697a-7dea-4662-8da4-c2618945d160" (UID: "d0c9697a-7dea-4662-8da4-c2618945d160"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:01:40.242260 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.241943 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-workload-socket" pod "d0c9697a-7dea-4662-8da4-c2618945d160" (UID: "d0c9697a-7dea-4662-8da4-c2618945d160"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:01:40.242964 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.242935 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-istio-envoy" pod "d0c9697a-7dea-4662-8da4-c2618945d160" (UID: "d0c9697a-7dea-4662-8da4-c2618945d160"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:01:40.244297 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.244257 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0c9697a-7dea-4662-8da4-c2618945d160-istio-podinfo" pod "d0c9697a-7dea-4662-8da4-c2618945d160" (UID: "d0c9697a-7dea-4662-8da4-c2618945d160"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:01:40.245655 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.245628 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c9697a-7dea-4662-8da4-c2618945d160-kube-api-access-ffg47" pod "d0c9697a-7dea-4662-8da4-c2618945d160" (UID: "d0c9697a-7dea-4662-8da4-c2618945d160"). InnerVolumeSpecName "kube-api-access-ffg47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:01:40.245752 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.245679 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c9697a-7dea-4662-8da4-c2618945d160-istio-token" pod "d0c9697a-7dea-4662-8da4-c2618945d160" (UID: "d0c9697a-7dea-4662-8da4-c2618945d160"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:01:40.342168 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.342073 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffg47\" (UniqueName: \"kubernetes.io/projected/d0c9697a-7dea-4662-8da4-c2618945d160-kube-api-access-ffg47\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:01:40.342168 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.342106 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c9697a-7dea-4662-8da4-c2618945d160-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:01:40.342168 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.342116 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:01:40.342168 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.342127 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c9697a-7dea-4662-8da4-c2618945d160-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:01:40.342168 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.342136 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:01:40.342168 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.342146 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c9697a-7dea-4662-8da4-c2618945d160-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:01:40.342168 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.342155 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:01:40.342168 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.342164 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c9697a-7dea-4662-8da4-c2618945d160-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:01:40.474891 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.474855 4951 generic.go:356] "Generic (PLEG): container finished" podID="d0c9697a-7dea-4662-8da4-c2618945d160" containerID="96f82bd1c8cc328ddf780900ba3326e9bf8394f6ae1ad18a83f832f3b8238e90" exitCode=0 Oct 30 08:01:40.475118 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.474913 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" Oct 30 08:01:40.475118 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.474931 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" event={"ID":"d0c9697a-7dea-4662-8da4-c2618945d160","Type":"ContainerDied","Data":"96f82bd1c8cc328ddf780900ba3326e9bf8394f6ae1ad18a83f832f3b8238e90"} Oct 30 08:01:40.475118 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.475000 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p" event={"ID":"d0c9697a-7dea-4662-8da4-c2618945d160","Type":"ContainerDied","Data":"158308feb10bee4ef6af6de8442674ba0f33b2fa34e42b066bda34421c25b5e1"} Oct 30 08:01:40.475118 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.475025 4951 scope.go:122] "RemoveContainer" containerID="96f82bd1c8cc328ddf780900ba3326e9bf8394f6ae1ad18a83f832f3b8238e90" Oct 30 08:01:40.495029 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.495001 4951 scope.go:122] "RemoveContainer" containerID="4c0dcfe40ef7c95e5594b253f38058adc8af6f1c0156d183282f1d07aa052521" Oct 30 08:01:40.506893 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.506871 4951 scope.go:122] "RemoveContainer" containerID="96f82bd1c8cc328ddf780900ba3326e9bf8394f6ae1ad18a83f832f3b8238e90" Oct 30 08:01:40.507239 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:01:40.507218 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96f82bd1c8cc328ddf780900ba3326e9bf8394f6ae1ad18a83f832f3b8238e90\": container with ID starting with 96f82bd1c8cc328ddf780900ba3326e9bf8394f6ae1ad18a83f832f3b8238e90 not found: ID does not exist" containerID="96f82bd1c8cc328ddf780900ba3326e9bf8394f6ae1ad18a83f832f3b8238e90" Oct 30 08:01:40.507304 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.507248 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f82bd1c8cc328ddf780900ba3326e9bf8394f6ae1ad18a83f832f3b8238e90"} err="failed to get container status \"96f82bd1c8cc328ddf780900ba3326e9bf8394f6ae1ad18a83f832f3b8238e90\": rpc error: code = NotFound desc = could not find container \"96f82bd1c8cc328ddf780900ba3326e9bf8394f6ae1ad18a83f832f3b8238e90\": container with ID starting with 96f82bd1c8cc328ddf780900ba3326e9bf8394f6ae1ad18a83f832f3b8238e90 not found: ID does not exist" Oct 30 08:01:40.507304 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.507266 4951 scope.go:122] "RemoveContainer" containerID="4c0dcfe40ef7c95e5594b253f38058adc8af6f1c0156d183282f1d07aa052521" Oct 30 08:01:40.507918 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:01:40.507886 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c0dcfe40ef7c95e5594b253f38058adc8af6f1c0156d183282f1d07aa052521\": container with ID starting with 4c0dcfe40ef7c95e5594b253f38058adc8af6f1c0156d183282f1d07aa052521 not found: ID does not exist" containerID="4c0dcfe40ef7c95e5594b253f38058adc8af6f1c0156d183282f1d07aa052521" Oct 30 08:01:40.508032 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.507925 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c0dcfe40ef7c95e5594b253f38058adc8af6f1c0156d183282f1d07aa052521"} err="failed to get container status \"4c0dcfe40ef7c95e5594b253f38058adc8af6f1c0156d183282f1d07aa052521\": rpc error: code = NotFound desc = could not find container \"4c0dcfe40ef7c95e5594b253f38058adc8af6f1c0156d183282f1d07aa052521\": container with ID starting with 4c0dcfe40ef7c95e5594b253f38058adc8af6f1c0156d183282f1d07aa052521 not found: ID does not exist" Oct 30 08:01:40.548841 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.548806 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p"] Oct 30 08:01:40.552279 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:40.552253 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-mtd6p"] Oct 30 08:01:41.776369 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:01:41.776332 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0c9697a-7dea-4662-8da4-c2618945d160" path="/var/lib/kubelet/pods/d0c9697a-7dea-4662-8da4-c2618945d160/volumes" Oct 30 08:01:53.026353 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:01:53.026314 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 08:02:23.026006 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:02:23.025954 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 08:02:34.596935 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:34.596890 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz"] Oct 30 08:02:34.597461 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:34.597189 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" podUID="e1304cf8-fc22-48db-acce-17514c25cf9d" containerName="istio-proxy" containerID="cri-o://0da3881c82a73b08b3e9def617785842b367756f987dcf7bce4100d3ea5b6d31" gracePeriod=30 Oct 30 08:02:39.894900 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:39.894874 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 08:02:39.974182 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:39.971752 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1304cf8-fc22-48db-acce-17514c25cf9d-kube-api-access-dpjg8\" (UniqueName: \"kubernetes.io/projected/e1304cf8-fc22-48db-acce-17514c25cf9d-kube-api-access-dpjg8\") pod \"e1304cf8-fc22-48db-acce-17514c25cf9d\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " Oct 30 08:02:39.974182 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:39.971828 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-credential-socket\") pod \"e1304cf8-fc22-48db-acce-17514c25cf9d\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " Oct 30 08:02:39.974182 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:39.971927 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-workload-socket\") pod \"e1304cf8-fc22-48db-acce-17514c25cf9d\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " Oct 30 08:02:39.974182 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:39.972069 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-istio-envoy\") pod \"e1304cf8-fc22-48db-acce-17514c25cf9d\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " Oct 30 08:02:39.974182 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:39.972128 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-istio-data\") pod \"e1304cf8-fc22-48db-acce-17514c25cf9d\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " Oct 30 08:02:39.974182 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:39.972178 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1304cf8-fc22-48db-acce-17514c25cf9d-istio-token\" (UniqueName: \"kubernetes.io/projected/e1304cf8-fc22-48db-acce-17514c25cf9d-istio-token\") pod \"e1304cf8-fc22-48db-acce-17514c25cf9d\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " Oct 30 08:02:39.974182 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:39.972262 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1304cf8-fc22-48db-acce-17514c25cf9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1304cf8-fc22-48db-acce-17514c25cf9d-istiod-ca-cert\") pod \"e1304cf8-fc22-48db-acce-17514c25cf9d\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " Oct 30 08:02:39.974182 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:39.972341 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1304cf8-fc22-48db-acce-17514c25cf9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1304cf8-fc22-48db-acce-17514c25cf9d-istio-podinfo\") pod \"e1304cf8-fc22-48db-acce-17514c25cf9d\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " Oct 30 08:02:39.974182 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:39.972437 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-workload-certs\") pod \"e1304cf8-fc22-48db-acce-17514c25cf9d\" (UID: \"e1304cf8-fc22-48db-acce-17514c25cf9d\") " Oct 30 08:02:39.974182 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:39.973869 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-workload-certs" pod "e1304cf8-fc22-48db-acce-17514c25cf9d" (UID: "e1304cf8-fc22-48db-acce-17514c25cf9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:02:39.980664 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:39.980624 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1304cf8-fc22-48db-acce-17514c25cf9d-kube-api-access-dpjg8" pod "e1304cf8-fc22-48db-acce-17514c25cf9d" (UID: "e1304cf8-fc22-48db-acce-17514c25cf9d"). InnerVolumeSpecName "kube-api-access-dpjg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:02:39.980909 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:39.980883 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-credential-socket" pod "e1304cf8-fc22-48db-acce-17514c25cf9d" (UID: "e1304cf8-fc22-48db-acce-17514c25cf9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:02:39.981276 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:39.981246 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-istio-data" pod "e1304cf8-fc22-48db-acce-17514c25cf9d" (UID: "e1304cf8-fc22-48db-acce-17514c25cf9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:02:39.981605 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:39.981547 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-workload-socket" pod "e1304cf8-fc22-48db-acce-17514c25cf9d" (UID: "e1304cf8-fc22-48db-acce-17514c25cf9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:02:39.981937 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:39.981913 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1304cf8-fc22-48db-acce-17514c25cf9d-istiod-ca-cert" pod "e1304cf8-fc22-48db-acce-17514c25cf9d" (UID: "e1304cf8-fc22-48db-acce-17514c25cf9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:02:39.992052 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:39.991749 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1304cf8-fc22-48db-acce-17514c25cf9d-istio-podinfo" pod "e1304cf8-fc22-48db-acce-17514c25cf9d" (UID: "e1304cf8-fc22-48db-acce-17514c25cf9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:02:39.992779 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:39.992745 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1304cf8-fc22-48db-acce-17514c25cf9d-istio-token" pod "e1304cf8-fc22-48db-acce-17514c25cf9d" (UID: "e1304cf8-fc22-48db-acce-17514c25cf9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:02:39.993068 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:39.993023 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-istio-envoy" pod "e1304cf8-fc22-48db-acce-17514c25cf9d" (UID: "e1304cf8-fc22-48db-acce-17514c25cf9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:02:40.001360 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:40.001315 4951 generic.go:356] "Generic (PLEG): container finished" podID="e1304cf8-fc22-48db-acce-17514c25cf9d" containerID="0da3881c82a73b08b3e9def617785842b367756f987dcf7bce4100d3ea5b6d31" exitCode=0 Oct 30 08:02:40.001601 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:40.001448 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" event={"ID":"e1304cf8-fc22-48db-acce-17514c25cf9d","Type":"ContainerDied","Data":"0da3881c82a73b08b3e9def617785842b367756f987dcf7bce4100d3ea5b6d31"} Oct 30 08:02:40.001601 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:40.001539 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" event={"ID":"e1304cf8-fc22-48db-acce-17514c25cf9d","Type":"ContainerDied","Data":"6a5cfdb780d74637336a72babbee10f63831b571ba6eef72f3f9e4638b61839b"} Oct 30 08:02:40.001756 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:40.001607 4951 scope.go:122] "RemoveContainer" containerID="0da3881c82a73b08b3e9def617785842b367756f987dcf7bce4100d3ea5b6d31" Oct 30 08:02:40.001831 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:40.001809 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz" Oct 30 08:02:40.022779 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:40.022753 4951 scope.go:122] "RemoveContainer" containerID="36d54d722f8f8b476a866fbab288a0b6c294c4bc6363a856345867f3360517d4" Oct 30 08:02:40.035553 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:40.035528 4951 scope.go:122] "RemoveContainer" containerID="0da3881c82a73b08b3e9def617785842b367756f987dcf7bce4100d3ea5b6d31" Oct 30 08:02:40.036074 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:02:40.036040 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0da3881c82a73b08b3e9def617785842b367756f987dcf7bce4100d3ea5b6d31\": container with ID starting with 0da3881c82a73b08b3e9def617785842b367756f987dcf7bce4100d3ea5b6d31 not found: ID does not exist" containerID="0da3881c82a73b08b3e9def617785842b367756f987dcf7bce4100d3ea5b6d31" Oct 30 08:02:40.036211 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:40.036084 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0da3881c82a73b08b3e9def617785842b367756f987dcf7bce4100d3ea5b6d31"} err="failed to get container status \"0da3881c82a73b08b3e9def617785842b367756f987dcf7bce4100d3ea5b6d31\": rpc error: code = NotFound desc = could not find container \"0da3881c82a73b08b3e9def617785842b367756f987dcf7bce4100d3ea5b6d31\": container with ID starting with 0da3881c82a73b08b3e9def617785842b367756f987dcf7bce4100d3ea5b6d31 not found: ID does not exist" Oct 30 08:02:40.036211 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:40.036109 4951 scope.go:122] "RemoveContainer" containerID="36d54d722f8f8b476a866fbab288a0b6c294c4bc6363a856345867f3360517d4" Oct 30 08:02:40.036479 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:02:40.036456 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d54d722f8f8b476a866fbab288a0b6c294c4bc6363a856345867f3360517d4\": container with ID starting with 36d54d722f8f8b476a866fbab288a0b6c294c4bc6363a856345867f3360517d4 not found: ID does not exist" containerID="36d54d722f8f8b476a866fbab288a0b6c294c4bc6363a856345867f3360517d4" Oct 30 08:02:40.036547 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:40.036490 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d54d722f8f8b476a866fbab288a0b6c294c4bc6363a856345867f3360517d4"} err="failed to get container status \"36d54d722f8f8b476a866fbab288a0b6c294c4bc6363a856345867f3360517d4\": rpc error: code = NotFound desc = could not find container \"36d54d722f8f8b476a866fbab288a0b6c294c4bc6363a856345867f3360517d4\": container with ID starting with 36d54d722f8f8b476a866fbab288a0b6c294c4bc6363a856345867f3360517d4 not found: ID does not exist" Oct 30 08:02:40.061051 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:40.061004 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz"] Oct 30 08:02:40.064712 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:40.064682 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-frpbz"] Oct 30 08:02:40.075020 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:40.074984 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1304cf8-fc22-48db-acce-17514c25cf9d-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:02:40.075020 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:40.075015 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:02:40.075020 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:40.075027 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpjg8\" (UniqueName: \"kubernetes.io/projected/e1304cf8-fc22-48db-acce-17514c25cf9d-kube-api-access-dpjg8\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:02:40.075272 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:40.075037 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:02:40.075272 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:40.075047 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:02:40.075272 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:40.075057 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:02:40.075272 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:40.075067 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1304cf8-fc22-48db-acce-17514c25cf9d-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:02:40.075272 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:40.075074 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1304cf8-fc22-48db-acce-17514c25cf9d-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:02:40.075272 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:40.075081 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1304cf8-fc22-48db-acce-17514c25cf9d-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:02:41.793641 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:41.792383 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1304cf8-fc22-48db-acce-17514c25cf9d" path="/var/lib/kubelet/pods/e1304cf8-fc22-48db-acce-17514c25cf9d/volumes" Oct 30 08:02:45.709587 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:45.709531 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-54f2b2-766bc666b6-ljwdv"] Oct 30 08:02:45.741825 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:45.741794 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-54f2b2-59f695d955-8fjrt"] Oct 30 08:02:50.709886 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:50.709832 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-54f2b2-766bc666b6-ljwdv" podUID="0f9d9e31-a0d8-4c02-93d9-5dbe88167315" containerName="payload-processing" containerID="cri-o://c63dc0ea79641a69e7fe894feae5a78c7a55b0ed7733ef55a1d820ebc59bc487" gracePeriod=25 Oct 30 08:02:50.742689 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:50.742632 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-54f2b2-59f695d955-8fjrt" podUID="4dd49e47-404e-4be7-8e30-26f4ceb2bc1b" containerName="payload-pre-processing" containerID="cri-o://38b10f6f06dec02173a46a80cb52a96444a302df0ad7d9639c03c81bc11b98e6" gracePeriod=25 Oct 30 08:02:51.086394 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:51.086365 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-54f2b2-766bc666b6-ljwdv" Oct 30 08:02:51.095381 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:51.095355 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f9d9e31-a0d8-4c02-93d9-5dbe88167315-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f9d9e31-a0d8-4c02-93d9-5dbe88167315-plugins-config-volume\") pod \"0f9d9e31-a0d8-4c02-93d9-5dbe88167315\" (UID: \"0f9d9e31-a0d8-4c02-93d9-5dbe88167315\") " Oct 30 08:02:51.095557 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:51.095536 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f9d9e31-a0d8-4c02-93d9-5dbe88167315-kube-api-access-bhzvh\" (UniqueName: \"kubernetes.io/projected/0f9d9e31-a0d8-4c02-93d9-5dbe88167315-kube-api-access-bhzvh\") pod \"0f9d9e31-a0d8-4c02-93d9-5dbe88167315\" (UID: \"0f9d9e31-a0d8-4c02-93d9-5dbe88167315\") " Oct 30 08:02:51.095774 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:51.095752 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f9d9e31-a0d8-4c02-93d9-5dbe88167315-plugins-config-volume" pod "0f9d9e31-a0d8-4c02-93d9-5dbe88167315" (UID: "0f9d9e31-a0d8-4c02-93d9-5dbe88167315"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:02:51.096144 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:51.096123 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f9d9e31-a0d8-4c02-93d9-5dbe88167315-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:02:51.098663 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:51.098641 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9d9e31-a0d8-4c02-93d9-5dbe88167315-kube-api-access-bhzvh" pod "0f9d9e31-a0d8-4c02-93d9-5dbe88167315" (UID: "0f9d9e31-a0d8-4c02-93d9-5dbe88167315"). InnerVolumeSpecName "kube-api-access-bhzvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:02:51.138618 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:51.134661 4951 generic.go:356] "Generic (PLEG): container finished" podID="0f9d9e31-a0d8-4c02-93d9-5dbe88167315" containerID="c63dc0ea79641a69e7fe894feae5a78c7a55b0ed7733ef55a1d820ebc59bc487" exitCode=0 Oct 30 08:02:51.138618 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:51.134803 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-54f2b2-766bc666b6-ljwdv" event={"ID":"0f9d9e31-a0d8-4c02-93d9-5dbe88167315","Type":"ContainerDied","Data":"c63dc0ea79641a69e7fe894feae5a78c7a55b0ed7733ef55a1d820ebc59bc487"} Oct 30 08:02:51.138618 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:51.134850 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-54f2b2-766bc666b6-ljwdv" event={"ID":"0f9d9e31-a0d8-4c02-93d9-5dbe88167315","Type":"ContainerDied","Data":"3b734cf016531a00e10ec9418dafae0983eb8caef8686d7d0c2e0d8eae94981a"} Oct 30 08:02:51.138618 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:51.134882 4951 scope.go:122] "RemoveContainer" containerID="c63dc0ea79641a69e7fe894feae5a78c7a55b0ed7733ef55a1d820ebc59bc487" Oct 30 08:02:51.138618 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:51.135043 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-54f2b2-766bc666b6-ljwdv" Oct 30 08:02:51.145381 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:51.145351 4951 generic.go:356] "Generic (PLEG): container finished" podID="4dd49e47-404e-4be7-8e30-26f4ceb2bc1b" containerID="38b10f6f06dec02173a46a80cb52a96444a302df0ad7d9639c03c81bc11b98e6" exitCode=0 Oct 30 08:02:51.145493 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:51.145420 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-54f2b2-59f695d955-8fjrt" event={"ID":"4dd49e47-404e-4be7-8e30-26f4ceb2bc1b","Type":"ContainerDied","Data":"38b10f6f06dec02173a46a80cb52a96444a302df0ad7d9639c03c81bc11b98e6"} Oct 30 08:02:51.147645 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:51.147625 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-54f2b2-59f695d955-8fjrt" Oct 30 08:02:51.157414 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:51.157396 4951 scope.go:122] "RemoveContainer" containerID="c63dc0ea79641a69e7fe894feae5a78c7a55b0ed7733ef55a1d820ebc59bc487" Oct 30 08:02:51.157861 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:02:51.157839 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c63dc0ea79641a69e7fe894feae5a78c7a55b0ed7733ef55a1d820ebc59bc487\": container with ID starting with c63dc0ea79641a69e7fe894feae5a78c7a55b0ed7733ef55a1d820ebc59bc487 not found: ID does not exist" containerID="c63dc0ea79641a69e7fe894feae5a78c7a55b0ed7733ef55a1d820ebc59bc487" Oct 30 08:02:51.157992 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:51.157970 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c63dc0ea79641a69e7fe894feae5a78c7a55b0ed7733ef55a1d820ebc59bc487"} err="failed to get container status \"c63dc0ea79641a69e7fe894feae5a78c7a55b0ed7733ef55a1d820ebc59bc487\": rpc error: code = NotFound desc = could not find container \"c63dc0ea79641a69e7fe894feae5a78c7a55b0ed7733ef55a1d820ebc59bc487\": container with ID starting with c63dc0ea79641a69e7fe894feae5a78c7a55b0ed7733ef55a1d820ebc59bc487 not found: ID does not exist" Oct 30 08:02:51.210356 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:51.199970 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-54f2b2-766bc666b6-ljwdv"] Oct 30 08:02:51.210356 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:51.202340 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dd49e47-404e-4be7-8e30-26f4ceb2bc1b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dd49e47-404e-4be7-8e30-26f4ceb2bc1b-plugins-config-volume\") pod \"4dd49e47-404e-4be7-8e30-26f4ceb2bc1b\" (UID: \"4dd49e47-404e-4be7-8e30-26f4ceb2bc1b\") " Oct 30 08:02:51.210356 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:51.202486 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dd49e47-404e-4be7-8e30-26f4ceb2bc1b-kube-api-access-64glc\" (UniqueName: \"kubernetes.io/projected/4dd49e47-404e-4be7-8e30-26f4ceb2bc1b-kube-api-access-64glc\") pod \"4dd49e47-404e-4be7-8e30-26f4ceb2bc1b\" (UID: \"4dd49e47-404e-4be7-8e30-26f4ceb2bc1b\") " Oct 30 08:02:51.210356 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:51.203122 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhzvh\" (UniqueName: \"kubernetes.io/projected/0f9d9e31-a0d8-4c02-93d9-5dbe88167315-kube-api-access-bhzvh\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:02:51.210356 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:51.204010 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dd49e47-404e-4be7-8e30-26f4ceb2bc1b-plugins-config-volume" pod "4dd49e47-404e-4be7-8e30-26f4ceb2bc1b" (UID: "4dd49e47-404e-4be7-8e30-26f4ceb2bc1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:02:51.210356 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:51.210034 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-54f2b2-766bc666b6-ljwdv"] Oct 30 08:02:51.210835 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:51.210653 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd49e47-404e-4be7-8e30-26f4ceb2bc1b-kube-api-access-64glc" pod "4dd49e47-404e-4be7-8e30-26f4ceb2bc1b" (UID: "4dd49e47-404e-4be7-8e30-26f4ceb2bc1b"). InnerVolumeSpecName "kube-api-access-64glc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:02:51.304499 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:51.304405 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dd49e47-404e-4be7-8e30-26f4ceb2bc1b-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:02:51.304499 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:51.304442 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64glc\" (UniqueName: \"kubernetes.io/projected/4dd49e47-404e-4be7-8e30-26f4ceb2bc1b-kube-api-access-64glc\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:02:51.775566 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:51.775533 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f9d9e31-a0d8-4c02-93d9-5dbe88167315" path="/var/lib/kubelet/pods/0f9d9e31-a0d8-4c02-93d9-5dbe88167315/volumes" Oct 30 08:02:52.155412 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:52.155379 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-54f2b2-59f695d955-8fjrt" Oct 30 08:02:52.155412 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:52.155390 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-54f2b2-59f695d955-8fjrt" event={"ID":"4dd49e47-404e-4be7-8e30-26f4ceb2bc1b","Type":"ContainerDied","Data":"1f17dce412acaf3c46e2e106071d8eb93f83bea78b9f9cbfa894c4de4d7271f5"} Oct 30 08:02:52.155727 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:52.155468 4951 scope.go:122] "RemoveContainer" containerID="38b10f6f06dec02173a46a80cb52a96444a302df0ad7d9639c03c81bc11b98e6" Oct 30 08:02:52.212078 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:52.212043 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-54f2b2-59f695d955-8fjrt"] Oct 30 08:02:52.224302 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:52.224274 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-54f2b2-59f695d955-8fjrt"] Oct 30 08:02:53.025795 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:02:53.025760 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 08:02:53.774333 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:53.774297 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dd49e47-404e-4be7-8e30-26f4ceb2bc1b" path="/var/lib/kubelet/pods/4dd49e47-404e-4be7-8e30-26f4ceb2bc1b/volumes" Oct 30 08:02:58.820486 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:58.820449 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4"] Oct 30 08:02:58.821049 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:02:58.820759 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" podUID="ef44a41b-a51b-49e4-aa7d-ae5ece873299" containerName="istio-proxy" containerID="cri-o://8a9b08338a6ad1e49c6765e87338eaa4ce72fb71c5a6d3d8477aa100251f515f" gracePeriod=30 Oct 30 08:03:04.122954 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.122920 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 08:03:04.244942 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.244901 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istio-podinfo\") pod \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " Oct 30 08:03:04.245138 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.244960 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istio-data\") pod \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " Oct 30 08:03:04.245138 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.245078 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istio-envoy\") pod \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " Oct 30 08:03:04.245138 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.245132 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef44a41b-a51b-49e4-aa7d-ae5ece873299-kube-api-access-qncbv\" (UniqueName: \"kubernetes.io/projected/ef44a41b-a51b-49e4-aa7d-ae5ece873299-kube-api-access-qncbv\") pod \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " Oct 30 08:03:04.245351 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.245292 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istio-data" pod "ef44a41b-a51b-49e4-aa7d-ae5ece873299" (UID: "ef44a41b-a51b-49e4-aa7d-ae5ece873299"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:03:04.245351 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.245299 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istiod-ca-cert\") pod \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " Oct 30 08:03:04.245475 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.245394 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-workload-certs\") pod \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " Oct 30 08:03:04.245530 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.245475 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istio-token\" (UniqueName: \"kubernetes.io/projected/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istio-token\") pod \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " Oct 30 08:03:04.245613 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.245524 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-workload-socket\") pod \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " Oct 30 08:03:04.245613 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.245605 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-credential-socket\") pod \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\" (UID: \"ef44a41b-a51b-49e4-aa7d-ae5ece873299\") " Oct 30 08:03:04.245790 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.245759 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istiod-ca-cert" pod "ef44a41b-a51b-49e4-aa7d-ae5ece873299" (UID: "ef44a41b-a51b-49e4-aa7d-ae5ece873299"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:03:04.245886 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.245864 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-workload-socket" pod "ef44a41b-a51b-49e4-aa7d-ae5ece873299" (UID: "ef44a41b-a51b-49e4-aa7d-ae5ece873299"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:03:04.246054 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.246027 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-workload-certs" pod "ef44a41b-a51b-49e4-aa7d-ae5ece873299" (UID: "ef44a41b-a51b-49e4-aa7d-ae5ece873299"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:03:04.246054 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.246042 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-credential-socket" pod "ef44a41b-a51b-49e4-aa7d-ae5ece873299" (UID: "ef44a41b-a51b-49e4-aa7d-ae5ece873299"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:03:04.246714 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.246620 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:04.246714 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.246648 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:04.246714 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.246661 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:04.246714 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.246676 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:04.246714 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.246690 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:04.248254 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.248219 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istio-envoy" pod "ef44a41b-a51b-49e4-aa7d-ae5ece873299" (UID: "ef44a41b-a51b-49e4-aa7d-ae5ece873299"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:03:04.248354 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.248282 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istio-token" pod "ef44a41b-a51b-49e4-aa7d-ae5ece873299" (UID: "ef44a41b-a51b-49e4-aa7d-ae5ece873299"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:03:04.248694 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.248661 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istio-podinfo" pod "ef44a41b-a51b-49e4-aa7d-ae5ece873299" (UID: "ef44a41b-a51b-49e4-aa7d-ae5ece873299"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:03:04.249184 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.249159 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef44a41b-a51b-49e4-aa7d-ae5ece873299-kube-api-access-qncbv" pod "ef44a41b-a51b-49e4-aa7d-ae5ece873299" (UID: "ef44a41b-a51b-49e4-aa7d-ae5ece873299"). InnerVolumeSpecName "kube-api-access-qncbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:03:04.280024 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.279991 4951 generic.go:356] "Generic (PLEG): container finished" podID="ef44a41b-a51b-49e4-aa7d-ae5ece873299" containerID="8a9b08338a6ad1e49c6765e87338eaa4ce72fb71c5a6d3d8477aa100251f515f" exitCode=0 Oct 30 08:03:04.280212 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.280058 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" event={"ID":"ef44a41b-a51b-49e4-aa7d-ae5ece873299","Type":"ContainerDied","Data":"8a9b08338a6ad1e49c6765e87338eaa4ce72fb71c5a6d3d8477aa100251f515f"} Oct 30 08:03:04.280212 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.280098 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" event={"ID":"ef44a41b-a51b-49e4-aa7d-ae5ece873299","Type":"ContainerDied","Data":"19ff653ace0b35c2476fd250ca340a354c0acb0fe6c80d0cb229e23cf57f694b"} Oct 30 08:03:04.280212 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.280120 4951 scope.go:122] "RemoveContainer" containerID="8a9b08338a6ad1e49c6765e87338eaa4ce72fb71c5a6d3d8477aa100251f515f" Oct 30 08:03:04.280405 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.280257 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4" Oct 30 08:03:04.295404 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.295377 4951 scope.go:122] "RemoveContainer" containerID="8a9b08338a6ad1e49c6765e87338eaa4ce72fb71c5a6d3d8477aa100251f515f" Oct 30 08:03:04.295842 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:03:04.295815 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a9b08338a6ad1e49c6765e87338eaa4ce72fb71c5a6d3d8477aa100251f515f\": container with ID starting with 8a9b08338a6ad1e49c6765e87338eaa4ce72fb71c5a6d3d8477aa100251f515f not found: ID does not exist" containerID="8a9b08338a6ad1e49c6765e87338eaa4ce72fb71c5a6d3d8477aa100251f515f" Oct 30 08:03:04.295955 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.295854 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a9b08338a6ad1e49c6765e87338eaa4ce72fb71c5a6d3d8477aa100251f515f"} err="failed to get container status \"8a9b08338a6ad1e49c6765e87338eaa4ce72fb71c5a6d3d8477aa100251f515f\": rpc error: code = NotFound desc = could not find container \"8a9b08338a6ad1e49c6765e87338eaa4ce72fb71c5a6d3d8477aa100251f515f\": container with ID starting with 8a9b08338a6ad1e49c6765e87338eaa4ce72fb71c5a6d3d8477aa100251f515f not found: ID does not exist" Oct 30 08:03:04.338791 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.338749 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4"] Oct 30 08:03:04.342477 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.342450 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-4rqq4"] Oct 30 08:03:04.347969 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.347946 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:04.347969 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.347972 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qncbv\" (UniqueName: \"kubernetes.io/projected/ef44a41b-a51b-49e4-aa7d-ae5ece873299-kube-api-access-qncbv\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:04.348180 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.347983 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:04.348180 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:04.347997 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef44a41b-a51b-49e4-aa7d-ae5ece873299-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:05.774169 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:05.774130 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef44a41b-a51b-49e4-aa7d-ae5ece873299" path="/var/lib/kubelet/pods/ef44a41b-a51b-49e4-aa7d-ae5ece873299/volumes" Oct 30 08:03:13.449026 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:13.448981 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv"] Oct 30 08:03:13.449543 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:13.449271 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv" podUID="a568debb-3a25-4cb8-a29c-6cb333299f02" containerName="maas-api" containerID="cri-o://33c0248966b61e21e362840c835a9885ca5d86ffd671bf84178ba9aff78f9c72" gracePeriod=30 Oct 30 08:03:13.845865 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:13.845833 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv" Oct 30 08:03:13.950455 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:13.941003 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a568debb-3a25-4cb8-a29c-6cb333299f02-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a568debb-3a25-4cb8-a29c-6cb333299f02-metrics-certs\") pod \"a568debb-3a25-4cb8-a29c-6cb333299f02\" (UID: \"a568debb-3a25-4cb8-a29c-6cb333299f02\") " Oct 30 08:03:13.950455 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:13.941092 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a568debb-3a25-4cb8-a29c-6cb333299f02-kube-api-access-q29zw\" (UniqueName: \"kubernetes.io/projected/a568debb-3a25-4cb8-a29c-6cb333299f02-kube-api-access-q29zw\") pod \"a568debb-3a25-4cb8-a29c-6cb333299f02\" (UID: \"a568debb-3a25-4cb8-a29c-6cb333299f02\") " Oct 30 08:03:13.950455 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:13.941169 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a568debb-3a25-4cb8-a29c-6cb333299f02-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a568debb-3a25-4cb8-a29c-6cb333299f02-maas-api-tls\") pod \"a568debb-3a25-4cb8-a29c-6cb333299f02\" (UID: \"a568debb-3a25-4cb8-a29c-6cb333299f02\") " Oct 30 08:03:13.950455 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:13.941274 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a568debb-3a25-4cb8-a29c-6cb333299f02-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a568debb-3a25-4cb8-a29c-6cb333299f02-openshift-service-ca\") pod \"a568debb-3a25-4cb8-a29c-6cb333299f02\" (UID: \"a568debb-3a25-4cb8-a29c-6cb333299f02\") " Oct 30 08:03:13.970622 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:13.954339 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a568debb-3a25-4cb8-a29c-6cb333299f02-metrics-certs" pod "a568debb-3a25-4cb8-a29c-6cb333299f02" (UID: "a568debb-3a25-4cb8-a29c-6cb333299f02"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:03:13.970622 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:13.961742 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a568debb-3a25-4cb8-a29c-6cb333299f02-kube-api-access-q29zw" pod "a568debb-3a25-4cb8-a29c-6cb333299f02" (UID: "a568debb-3a25-4cb8-a29c-6cb333299f02"). InnerVolumeSpecName "kube-api-access-q29zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:03:13.971274 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:13.971242 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a568debb-3a25-4cb8-a29c-6cb333299f02-openshift-service-ca" pod "a568debb-3a25-4cb8-a29c-6cb333299f02" (UID: "a568debb-3a25-4cb8-a29c-6cb333299f02"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:03:13.982901 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:13.982158 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a568debb-3a25-4cb8-a29c-6cb333299f02-maas-api-tls" pod "a568debb-3a25-4cb8-a29c-6cb333299f02" (UID: "a568debb-3a25-4cb8-a29c-6cb333299f02"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:03:14.044131 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:14.044085 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a568debb-3a25-4cb8-a29c-6cb333299f02-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:14.044131 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:14.044131 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a568debb-3a25-4cb8-a29c-6cb333299f02-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:14.044411 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:14.044146 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q29zw\" (UniqueName: \"kubernetes.io/projected/a568debb-3a25-4cb8-a29c-6cb333299f02-kube-api-access-q29zw\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:14.044411 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:14.044161 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a568debb-3a25-4cb8-a29c-6cb333299f02-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:14.377780 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:14.377743 4951 generic.go:356] "Generic (PLEG): container finished" podID="a568debb-3a25-4cb8-a29c-6cb333299f02" containerID="33c0248966b61e21e362840c835a9885ca5d86ffd671bf84178ba9aff78f9c72" exitCode=0 Oct 30 08:03:14.377960 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:14.377800 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv" Oct 30 08:03:14.377960 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:14.377843 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv" event={"ID":"a568debb-3a25-4cb8-a29c-6cb333299f02","Type":"ContainerDied","Data":"33c0248966b61e21e362840c835a9885ca5d86ffd671bf84178ba9aff78f9c72"} Oct 30 08:03:14.377960 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:14.377915 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv" event={"ID":"a568debb-3a25-4cb8-a29c-6cb333299f02","Type":"ContainerDied","Data":"5b53a93d9b8c7530e428ed60ebae776d65c3477b324f9e39acd45588cb9dfa4a"} Oct 30 08:03:14.377960 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:14.377948 4951 scope.go:122] "RemoveContainer" containerID="33c0248966b61e21e362840c835a9885ca5d86ffd671bf84178ba9aff78f9c72" Oct 30 08:03:14.394255 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:14.394230 4951 scope.go:122] "RemoveContainer" containerID="33c0248966b61e21e362840c835a9885ca5d86ffd671bf84178ba9aff78f9c72" Oct 30 08:03:14.394633 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:03:14.394605 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33c0248966b61e21e362840c835a9885ca5d86ffd671bf84178ba9aff78f9c72\": container with ID starting with 33c0248966b61e21e362840c835a9885ca5d86ffd671bf84178ba9aff78f9c72 not found: ID does not exist" containerID="33c0248966b61e21e362840c835a9885ca5d86ffd671bf84178ba9aff78f9c72" Oct 30 08:03:14.394755 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:14.394639 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33c0248966b61e21e362840c835a9885ca5d86ffd671bf84178ba9aff78f9c72"} err="failed to get container status \"33c0248966b61e21e362840c835a9885ca5d86ffd671bf84178ba9aff78f9c72\": rpc error: code = NotFound desc = could not find container \"33c0248966b61e21e362840c835a9885ca5d86ffd671bf84178ba9aff78f9c72\": container with ID starting with 33c0248966b61e21e362840c835a9885ca5d86ffd671bf84178ba9aff78f9c72 not found: ID does not exist" Oct 30 08:03:14.437680 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:14.437651 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv"] Oct 30 08:03:14.443419 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:14.443390 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv"] Oct 30 08:03:15.841848 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:15.841544 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a568debb-3a25-4cb8-a29c-6cb333299f02" path="/var/lib/kubelet/pods/a568debb-3a25-4cb8-a29c-6cb333299f02/volumes" Oct 30 08:03:16.794302 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:16.794256 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-54f2b2-696fcb5797-dr4jv" podUID="a568debb-3a25-4cb8-a29c-6cb333299f02" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.211:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 08:03:21.755845 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:21.755792 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x"] Oct 30 08:03:21.756403 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:21.756182 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" podUID="8562e54e-5cb6-4805-ae61-afc7b1d5c0c6" containerName="istio-proxy" containerID="cri-o://32df6e73728e70dbf4bb3d0c6008a7183be84aac4919ea08498ad029d6f4c7d8" gracePeriod=30 Oct 30 08:03:23.023656 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:03:23.023615 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 08:03:27.055088 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.055017 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 08:03:27.146558 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.146519 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istiod-ca-cert\") pod \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " Oct 30 08:03:27.146793 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.146637 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istio-token\" (UniqueName: \"kubernetes.io/projected/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istio-token\") pod \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " Oct 30 08:03:27.146793 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.146687 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-workload-socket\") pod \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " Oct 30 08:03:27.146793 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.146734 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-kube-api-access-cvnlf\" (UniqueName: \"kubernetes.io/projected/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-kube-api-access-cvnlf\") pod \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " Oct 30 08:03:27.146973 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.146800 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istio-podinfo\") pod \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " Oct 30 08:03:27.146973 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.146871 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istiod-ca-cert" pod "8562e54e-5cb6-4805-ae61-afc7b1d5c0c6" (UID: "8562e54e-5cb6-4805-ae61-afc7b1d5c0c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:03:27.146973 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.146894 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-credential-socket\") pod \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " Oct 30 08:03:27.146973 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.146929 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istio-envoy\") pod \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " Oct 30 08:03:27.146973 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.146959 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-workload-certs\") pod \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " Oct 30 08:03:27.147230 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.147053 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istio-data\") pod \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\" (UID: \"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6\") " Oct 30 08:03:27.147630 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.147603 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-credential-socket" pod "8562e54e-5cb6-4805-ae61-afc7b1d5c0c6" (UID: "8562e54e-5cb6-4805-ae61-afc7b1d5c0c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:03:27.147840 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.147797 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:27.148080 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.148057 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-workload-socket" pod "8562e54e-5cb6-4805-ae61-afc7b1d5c0c6" (UID: "8562e54e-5cb6-4805-ae61-afc7b1d5c0c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:03:27.148479 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.148443 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-workload-certs" pod "8562e54e-5cb6-4805-ae61-afc7b1d5c0c6" (UID: "8562e54e-5cb6-4805-ae61-afc7b1d5c0c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:03:27.148862 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.148801 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istio-data" pod "8562e54e-5cb6-4805-ae61-afc7b1d5c0c6" (UID: "8562e54e-5cb6-4805-ae61-afc7b1d5c0c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:03:27.149731 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.149704 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-kube-api-access-cvnlf" pod "8562e54e-5cb6-4805-ae61-afc7b1d5c0c6" (UID: "8562e54e-5cb6-4805-ae61-afc7b1d5c0c6"). InnerVolumeSpecName "kube-api-access-cvnlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:03:27.149841 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.149751 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istio-token" pod "8562e54e-5cb6-4805-ae61-afc7b1d5c0c6" (UID: "8562e54e-5cb6-4805-ae61-afc7b1d5c0c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:03:27.150303 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.150274 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istio-podinfo" pod "8562e54e-5cb6-4805-ae61-afc7b1d5c0c6" (UID: "8562e54e-5cb6-4805-ae61-afc7b1d5c0c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:03:27.150892 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.150855 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istio-envoy" pod "8562e54e-5cb6-4805-ae61-afc7b1d5c0c6" (UID: "8562e54e-5cb6-4805-ae61-afc7b1d5c0c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:03:27.255452 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.255410 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:27.255452 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.255451 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:27.255452 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.255463 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:27.255817 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.255474 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:27.255817 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.255485 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:27.255817 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.255510 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvnlf\" (UniqueName: \"kubernetes.io/projected/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-kube-api-access-cvnlf\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:27.255817 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.255524 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:27.255817 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.255536 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:27.545306 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.545269 4951 generic.go:356] "Generic (PLEG): container finished" podID="8562e54e-5cb6-4805-ae61-afc7b1d5c0c6" containerID="32df6e73728e70dbf4bb3d0c6008a7183be84aac4919ea08498ad029d6f4c7d8" exitCode=0 Oct 30 08:03:27.545518 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.545325 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" Oct 30 08:03:27.545518 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.545359 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" event={"ID":"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6","Type":"ContainerDied","Data":"32df6e73728e70dbf4bb3d0c6008a7183be84aac4919ea08498ad029d6f4c7d8"} Oct 30 08:03:27.545518 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.545441 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x" event={"ID":"8562e54e-5cb6-4805-ae61-afc7b1d5c0c6","Type":"ContainerDied","Data":"bf3411f6363fa8db46d40e9705f509656da806cee14213bdf147da35220d46f9"} Oct 30 08:03:27.545518 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.545472 4951 scope.go:122] "RemoveContainer" containerID="32df6e73728e70dbf4bb3d0c6008a7183be84aac4919ea08498ad029d6f4c7d8" Oct 30 08:03:27.557411 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.557387 4951 scope.go:122] "RemoveContainer" containerID="298585ce83863ba43af616efbe835894bca8b711581bdc06ca7bff1d3ab975b7" Oct 30 08:03:27.568017 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.567992 4951 scope.go:122] "RemoveContainer" containerID="32df6e73728e70dbf4bb3d0c6008a7183be84aac4919ea08498ad029d6f4c7d8" Oct 30 08:03:27.568441 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:03:27.568408 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32df6e73728e70dbf4bb3d0c6008a7183be84aac4919ea08498ad029d6f4c7d8\": container with ID starting with 32df6e73728e70dbf4bb3d0c6008a7183be84aac4919ea08498ad029d6f4c7d8 not found: ID does not exist" containerID="32df6e73728e70dbf4bb3d0c6008a7183be84aac4919ea08498ad029d6f4c7d8" Oct 30 08:03:27.568602 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.568454 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32df6e73728e70dbf4bb3d0c6008a7183be84aac4919ea08498ad029d6f4c7d8"} err="failed to get container status \"32df6e73728e70dbf4bb3d0c6008a7183be84aac4919ea08498ad029d6f4c7d8\": rpc error: code = NotFound desc = could not find container \"32df6e73728e70dbf4bb3d0c6008a7183be84aac4919ea08498ad029d6f4c7d8\": container with ID starting with 32df6e73728e70dbf4bb3d0c6008a7183be84aac4919ea08498ad029d6f4c7d8 not found: ID does not exist" Oct 30 08:03:27.568602 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.568482 4951 scope.go:122] "RemoveContainer" containerID="298585ce83863ba43af616efbe835894bca8b711581bdc06ca7bff1d3ab975b7" Oct 30 08:03:27.568921 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:03:27.568892 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"298585ce83863ba43af616efbe835894bca8b711581bdc06ca7bff1d3ab975b7\": container with ID starting with 298585ce83863ba43af616efbe835894bca8b711581bdc06ca7bff1d3ab975b7 not found: ID does not exist" containerID="298585ce83863ba43af616efbe835894bca8b711581bdc06ca7bff1d3ab975b7" Oct 30 08:03:27.568968 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.568932 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"298585ce83863ba43af616efbe835894bca8b711581bdc06ca7bff1d3ab975b7"} err="failed to get container status \"298585ce83863ba43af616efbe835894bca8b711581bdc06ca7bff1d3ab975b7\": rpc error: code = NotFound desc = could not find container \"298585ce83863ba43af616efbe835894bca8b711581bdc06ca7bff1d3ab975b7\": container with ID starting with 298585ce83863ba43af616efbe835894bca8b711581bdc06ca7bff1d3ab975b7 not found: ID does not exist" Oct 30 08:03:27.605736 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.605702 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x"] Oct 30 08:03:27.621316 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.619910 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-54f2b2-openshift-default-6cfd9b66cc-97h6x"] Oct 30 08:03:27.773980 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:27.773940 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8562e54e-5cb6-4805-ae61-afc7b1d5c0c6" path="/var/lib/kubelet/pods/8562e54e-5cb6-4805-ae61-afc7b1d5c0c6/volumes" Oct 30 08:03:32.765355 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:32.765320 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-228571-5995bc74cb-5kvfx"] Oct 30 08:03:32.798488 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:32.798451 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-228571-676b499bc-k76j6"] Oct 30 08:03:35.809242 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.809197 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j"] Oct 30 08:03:35.810807 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.810780 4951 cpu_manager.go:414] "containerMap: removing container" podUID="8562e54e-5cb6-4805-ae61-afc7b1d5c0c6" containerName="istio-proxy" Oct 30 08:03:35.810807 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.810807 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8562e54e-5cb6-4805-ae61-afc7b1d5c0c6" containerName="istio-proxy" Oct 30 08:03:35.811010 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.810834 4951 cpu_manager.go:414] "containerMap: removing container" podUID="abf16ce5-dabd-44dd-a441-9b7059c28338" containerName="maas-api" Oct 30 08:03:35.811010 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.810843 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abf16ce5-dabd-44dd-a441-9b7059c28338" containerName="maas-api" Oct 30 08:03:35.811010 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.810866 4951 cpu_manager.go:414] "containerMap: removing container" podUID="ef44a41b-a51b-49e4-aa7d-ae5ece873299" containerName="istio-proxy" Oct 30 08:03:35.811010 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.810873 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef44a41b-a51b-49e4-aa7d-ae5ece873299" containerName="istio-proxy" Oct 30 08:03:35.811010 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.810890 4951 cpu_manager.go:414] "containerMap: removing container" podUID="d3ce0d73-117c-4309-9f1d-71a0a6c53299" containerName="cleanup" Oct 30 08:03:35.811010 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.810897 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3ce0d73-117c-4309-9f1d-71a0a6c53299" containerName="cleanup" Oct 30 08:03:35.811010 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.810923 4951 cpu_manager.go:414] "containerMap: removing container" podUID="d3ce0d73-117c-4309-9f1d-71a0a6c53299" containerName="cleanup" Oct 30 08:03:35.811010 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.810930 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3ce0d73-117c-4309-9f1d-71a0a6c53299" containerName="cleanup" Oct 30 08:03:35.811010 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.810964 4951 cpu_manager.go:414] "containerMap: removing container" podUID="8562e54e-5cb6-4805-ae61-afc7b1d5c0c6" containerName="istio-proxy" Oct 30 08:03:35.811010 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.810971 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8562e54e-5cb6-4805-ae61-afc7b1d5c0c6" containerName="istio-proxy" Oct 30 08:03:35.811010 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.810981 4951 cpu_manager.go:414] "containerMap: removing container" podUID="e1304cf8-fc22-48db-acce-17514c25cf9d" containerName="istio-proxy" Oct 30 08:03:35.811010 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.810987 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1304cf8-fc22-48db-acce-17514c25cf9d" containerName="istio-proxy" Oct 30 08:03:35.811010 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.811003 4951 cpu_manager.go:414] "containerMap: removing container" podUID="3933eee7-a582-4582-a7dc-23763490fad8" containerName="cleanup" Oct 30 08:03:35.811010 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.811010 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3933eee7-a582-4582-a7dc-23763490fad8" containerName="cleanup" Oct 30 08:03:35.811713 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.811031 4951 cpu_manager.go:414] "containerMap: removing container" podUID="0f9d9e31-a0d8-4c02-93d9-5dbe88167315" containerName="payload-processing" Oct 30 08:03:35.811713 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.811038 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f9d9e31-a0d8-4c02-93d9-5dbe88167315" containerName="payload-processing" Oct 30 08:03:35.811713 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.811064 4951 cpu_manager.go:414] "containerMap: removing container" podUID="cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2" containerName="payload-pre-processing" Oct 30 08:03:35.811713 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.811075 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2" containerName="payload-pre-processing" Oct 30 08:03:35.811713 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.811089 4951 cpu_manager.go:414] "containerMap: removing container" podUID="82220bd1-27c2-447d-967d-80032aa1480d" containerName="cleanup" Oct 30 08:03:35.811713 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.811095 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82220bd1-27c2-447d-967d-80032aa1480d" containerName="cleanup" Oct 30 08:03:35.811713 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.811134 4951 cpu_manager.go:414] "containerMap: removing container" podUID="d0c9697a-7dea-4662-8da4-c2618945d160" containerName="istio-proxy" Oct 30 08:03:35.811713 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.811143 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c9697a-7dea-4662-8da4-c2618945d160" containerName="istio-proxy" Oct 30 08:03:35.811713 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.811167 4951 cpu_manager.go:414] "containerMap: removing container" podUID="4dd49e47-404e-4be7-8e30-26f4ceb2bc1b" containerName="payload-pre-processing" Oct 30 08:03:35.811713 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.811174 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dd49e47-404e-4be7-8e30-26f4ceb2bc1b" containerName="payload-pre-processing" Oct 30 08:03:35.829727 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.829699 4951 cpu_manager.go:414] "containerMap: removing container" podUID="d0c9697a-7dea-4662-8da4-c2618945d160" containerName="istio-proxy" Oct 30 08:03:35.829978 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.829936 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c9697a-7dea-4662-8da4-c2618945d160" containerName="istio-proxy" Oct 30 08:03:35.830128 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.829992 4951 cpu_manager.go:414] "containerMap: removing container" podUID="1ad0084a-1d54-4b23-ac5c-99cbc8b6da24" containerName="payload-processing" Oct 30 08:03:35.830128 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.830001 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ad0084a-1d54-4b23-ac5c-99cbc8b6da24" containerName="payload-processing" Oct 30 08:03:35.830128 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.830018 4951 cpu_manager.go:414] "containerMap: removing container" podUID="a568debb-3a25-4cb8-a29c-6cb333299f02" containerName="maas-api" Oct 30 08:03:35.830128 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.830026 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a568debb-3a25-4cb8-a29c-6cb333299f02" containerName="maas-api" Oct 30 08:03:35.830128 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.830048 4951 cpu_manager.go:414] "containerMap: removing container" podUID="82220bd1-27c2-447d-967d-80032aa1480d" containerName="cleanup" Oct 30 08:03:35.830128 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.830056 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82220bd1-27c2-447d-967d-80032aa1480d" containerName="cleanup" Oct 30 08:03:35.830128 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.830086 4951 cpu_manager.go:414] "containerMap: removing container" podUID="3933eee7-a582-4582-a7dc-23763490fad8" containerName="cleanup" Oct 30 08:03:35.830128 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.830095 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3933eee7-a582-4582-a7dc-23763490fad8" containerName="cleanup" Oct 30 08:03:35.830128 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.830126 4951 cpu_manager.go:414] "containerMap: removing container" podUID="90e2dc87-6736-4b39-acfe-8db6ee096cd5" containerName="istio-proxy" Oct 30 08:03:35.830711 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.830133 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90e2dc87-6736-4b39-acfe-8db6ee096cd5" containerName="istio-proxy" Oct 30 08:03:35.830711 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.830502 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c9697a-7dea-4662-8da4-c2618945d160" containerName="istio-proxy" Oct 30 08:03:35.830711 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.830530 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ad0084a-1d54-4b23-ac5c-99cbc8b6da24" containerName="payload-processing" Oct 30 08:03:35.830711 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.830548 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="a568debb-3a25-4cb8-a29c-6cb333299f02" containerName="maas-api" Oct 30 08:03:35.830711 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.830558 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="3933eee7-a582-4582-a7dc-23763490fad8" containerName="cleanup" Oct 30 08:03:35.830711 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.830594 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="82220bd1-27c2-447d-967d-80032aa1480d" containerName="cleanup" Oct 30 08:03:35.830711 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.830606 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="3933eee7-a582-4582-a7dc-23763490fad8" containerName="cleanup" Oct 30 08:03:35.830711 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.830620 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="90e2dc87-6736-4b39-acfe-8db6ee096cd5" containerName="istio-proxy" Oct 30 08:03:35.830711 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.830645 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="8562e54e-5cb6-4805-ae61-afc7b1d5c0c6" containerName="istio-proxy" Oct 30 08:03:35.830711 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.830655 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="abf16ce5-dabd-44dd-a441-9b7059c28338" containerName="maas-api" Oct 30 08:03:35.830711 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.830679 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef44a41b-a51b-49e4-aa7d-ae5ece873299" containerName="istio-proxy" Oct 30 08:03:35.830711 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.830688 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3ce0d73-117c-4309-9f1d-71a0a6c53299" containerName="cleanup" Oct 30 08:03:35.830711 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.830699 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3ce0d73-117c-4309-9f1d-71a0a6c53299" containerName="cleanup" Oct 30 08:03:35.830711 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.830719 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="8562e54e-5cb6-4805-ae61-afc7b1d5c0c6" containerName="istio-proxy" Oct 30 08:03:35.831431 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.830733 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1304cf8-fc22-48db-acce-17514c25cf9d" containerName="istio-proxy" Oct 30 08:03:35.831431 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.830751 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f9d9e31-a0d8-4c02-93d9-5dbe88167315" containerName="payload-processing" Oct 30 08:03:35.831431 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.830763 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="cea9fef6-4eaf-451c-aee0-8d0d2c70c2a2" containerName="payload-pre-processing" Oct 30 08:03:35.831431 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.830779 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="82220bd1-27c2-447d-967d-80032aa1480d" containerName="cleanup" Oct 30 08:03:35.831431 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.830789 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1304cf8-fc22-48db-acce-17514c25cf9d" containerName="istio-proxy" Oct 30 08:03:35.831431 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.830810 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dd49e47-404e-4be7-8e30-26f4ceb2bc1b" containerName="payload-pre-processing" Oct 30 08:03:35.853404 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.852557 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j"] Oct 30 08:03:35.853404 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.852717 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:03:35.871608 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.871558 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-bd38cf-openshift-default-dockercfg-rphjj\"" Oct 30 08:03:35.892270 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.892226 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-workload-certs\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:03:35.892479 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.892283 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-istio-data\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:03:35.892479 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.892424 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a796d7df-446c-419e-be91-f479e9bfa8e2-istio-podinfo\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:03:35.892479 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.892469 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-istio-envoy\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:03:35.892814 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.892535 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a796d7df-446c-419e-be91-f479e9bfa8e2-istio-token\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:03:35.892814 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.892565 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm7rp\" (UniqueName: \"kubernetes.io/projected/a796d7df-446c-419e-be91-f479e9bfa8e2-kube-api-access-dm7rp\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:03:35.892944 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.892842 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a796d7df-446c-419e-be91-f479e9bfa8e2-istiod-ca-cert\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:03:35.892944 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.892903 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-workload-socket\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:03:35.893067 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.892958 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-credential-socket\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:03:35.995611 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.995546 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-workload-socket\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:03:35.996412 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.996384 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-credential-socket\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:03:35.997002 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.996982 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-workload-certs\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:03:35.997450 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.997433 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-istio-data\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:03:35.997927 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.997908 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a796d7df-446c-419e-be91-f479e9bfa8e2-istio-podinfo\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:03:35.998600 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.998561 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-istio-envoy\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:03:36.000281 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:36.000257 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a796d7df-446c-419e-be91-f479e9bfa8e2-istio-token\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:03:36.001063 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:36.000776 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm7rp\" (UniqueName: \"kubernetes.io/projected/a796d7df-446c-419e-be91-f479e9bfa8e2-kube-api-access-dm7rp\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:03:36.001461 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:36.001441 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a796d7df-446c-419e-be91-f479e9bfa8e2-istiod-ca-cert\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:03:36.002397 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.997378 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-workload-certs\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:03:36.002534 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.996861 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-credential-socket\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:03:36.002657 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.997814 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-istio-data\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:03:36.002760 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:35.996289 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-workload-socket\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:03:36.004029 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:36.002241 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a796d7df-446c-419e-be91-f479e9bfa8e2-istiod-ca-cert\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:03:36.056104 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:36.056066 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-istio-envoy\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:03:36.057240 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:36.057213 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a796d7df-446c-419e-be91-f479e9bfa8e2-istio-podinfo\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:03:36.058212 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:36.058163 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a796d7df-446c-419e-be91-f479e9bfa8e2-istio-token\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:03:36.087639 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:36.084197 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm7rp\" (UniqueName: \"kubernetes.io/projected/a796d7df-446c-419e-be91-f479e9bfa8e2-kube-api-access-dm7rp\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:03:36.177221 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:36.177179 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:03:36.508033 ip-10-0-132-173 kubenswrapper[4951]: W1030 08:03:36.507988 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda796d7df_446c_419e_be91_f479e9bfa8e2.slice/crio-5246d81fccd0df3b67b53ea055f312a6bdb49629de5f317de58ff6784c9b9008 WatchSource:0}: Error finding container 5246d81fccd0df3b67b53ea055f312a6bdb49629de5f317de58ff6784c9b9008: Status 404 returned error can't find the container with id 5246d81fccd0df3b67b53ea055f312a6bdb49629de5f317de58ff6784c9b9008 Oct 30 08:03:36.509214 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:36.509184 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j"] Oct 30 08:03:36.511011 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:36.510970 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 08:03:36.511171 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:36.511048 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 08:03:36.511171 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:36.511079 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 08:03:36.618267 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:36.618230 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" event={"ID":"a796d7df-446c-419e-be91-f479e9bfa8e2","Type":"ContainerStarted","Data":"6bacce6fd9a94862334d7f540bf1296875d0445f548e36b69c7cab628ce7f2ef"} Oct 30 08:03:36.618429 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:36.618289 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" event={"ID":"a796d7df-446c-419e-be91-f479e9bfa8e2","Type":"ContainerStarted","Data":"5246d81fccd0df3b67b53ea055f312a6bdb49629de5f317de58ff6784c9b9008"} Oct 30 08:03:36.668594 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:36.668273 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" podStartSLOduration=1.668259254 podStartE2EDuration="1.668259254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:03:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:03:36.664275984 +0000 UTC m=+4023.007323964" watchObservedRunningTime="2026-10-30 08:03:36.668259254 +0000 UTC m=+4023.011307218" Oct 30 08:03:37.178114 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:37.178073 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:03:37.179585 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:37.179546 4951 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Oct 30 08:03:37.179729 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:37.179619 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" podUID="a796d7df-446c-419e-be91-f479e9bfa8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Oct 30 08:03:37.765976 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:37.765936 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-228571-5995bc74cb-5kvfx" podUID="399020ed-d53c-41b2-8b3d-298647396c87" containerName="payload-processing" containerID="cri-o://763035ff78ae4ceeec5f0ec9862891754f440b7ea2ce90a6e4e9044b405f7131" gracePeriod=25 Oct 30 08:03:37.799424 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:37.799390 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-228571-676b499bc-k76j6" podUID="9c290e9a-af71-48e8-99c6-43056987a08d" containerName="payload-pre-processing" containerID="cri-o://11972b0894266cf0bda6b40d698fcaa63a51bfc34ae04ff127fdc891b8fa6faf" gracePeriod=25 Oct 30 08:03:38.177709 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.177678 4951 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Oct 30 08:03:38.177861 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.177733 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" podUID="a796d7df-446c-419e-be91-f479e9bfa8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Oct 30 08:03:38.183038 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.183015 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-228571-5995bc74cb-5kvfx" Oct 30 08:03:38.272347 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.270719 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/399020ed-d53c-41b2-8b3d-298647396c87-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/399020ed-d53c-41b2-8b3d-298647396c87-plugins-config-volume\") pod \"399020ed-d53c-41b2-8b3d-298647396c87\" (UID: \"399020ed-d53c-41b2-8b3d-298647396c87\") " Oct 30 08:03:38.272347 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.270814 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/399020ed-d53c-41b2-8b3d-298647396c87-kube-api-access-qrqgk\" (UniqueName: \"kubernetes.io/projected/399020ed-d53c-41b2-8b3d-298647396c87-kube-api-access-qrqgk\") pod \"399020ed-d53c-41b2-8b3d-298647396c87\" (UID: \"399020ed-d53c-41b2-8b3d-298647396c87\") " Oct 30 08:03:38.272347 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.271995 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/399020ed-d53c-41b2-8b3d-298647396c87-plugins-config-volume" pod "399020ed-d53c-41b2-8b3d-298647396c87" (UID: "399020ed-d53c-41b2-8b3d-298647396c87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:03:38.274175 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.274143 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/399020ed-d53c-41b2-8b3d-298647396c87-kube-api-access-qrqgk" pod "399020ed-d53c-41b2-8b3d-298647396c87" (UID: "399020ed-d53c-41b2-8b3d-298647396c87"). InnerVolumeSpecName "kube-api-access-qrqgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:03:38.311776 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.311751 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-228571-676b499bc-k76j6" Oct 30 08:03:38.372447 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.372414 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/399020ed-d53c-41b2-8b3d-298647396c87-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:38.372447 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.372443 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrqgk\" (UniqueName: \"kubernetes.io/projected/399020ed-d53c-41b2-8b3d-298647396c87-kube-api-access-qrqgk\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:38.473336 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.473230 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c290e9a-af71-48e8-99c6-43056987a08d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c290e9a-af71-48e8-99c6-43056987a08d-plugins-config-volume\") pod \"9c290e9a-af71-48e8-99c6-43056987a08d\" (UID: \"9c290e9a-af71-48e8-99c6-43056987a08d\") " Oct 30 08:03:38.473512 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.473386 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c290e9a-af71-48e8-99c6-43056987a08d-kube-api-access-cr5wx\" (UniqueName: \"kubernetes.io/projected/9c290e9a-af71-48e8-99c6-43056987a08d-kube-api-access-cr5wx\") pod \"9c290e9a-af71-48e8-99c6-43056987a08d\" (UID: \"9c290e9a-af71-48e8-99c6-43056987a08d\") " Oct 30 08:03:38.473691 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.473667 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c290e9a-af71-48e8-99c6-43056987a08d-plugins-config-volume" pod "9c290e9a-af71-48e8-99c6-43056987a08d" (UID: "9c290e9a-af71-48e8-99c6-43056987a08d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:03:38.474116 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.474093 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c290e9a-af71-48e8-99c6-43056987a08d-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:38.475745 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.475720 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c290e9a-af71-48e8-99c6-43056987a08d-kube-api-access-cr5wx" pod "9c290e9a-af71-48e8-99c6-43056987a08d" (UID: "9c290e9a-af71-48e8-99c6-43056987a08d"). InnerVolumeSpecName "kube-api-access-cr5wx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:03:38.575099 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.575056 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr5wx\" (UniqueName: \"kubernetes.io/projected/9c290e9a-af71-48e8-99c6-43056987a08d-kube-api-access-cr5wx\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:38.654049 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.654009 4951 generic.go:356] "Generic (PLEG): container finished" podID="9c290e9a-af71-48e8-99c6-43056987a08d" containerID="11972b0894266cf0bda6b40d698fcaa63a51bfc34ae04ff127fdc891b8fa6faf" exitCode=0 Oct 30 08:03:38.654261 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.654060 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-228571-676b499bc-k76j6" Oct 30 08:03:38.654261 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.654064 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-228571-676b499bc-k76j6" event={"ID":"9c290e9a-af71-48e8-99c6-43056987a08d","Type":"ContainerDied","Data":"11972b0894266cf0bda6b40d698fcaa63a51bfc34ae04ff127fdc891b8fa6faf"} Oct 30 08:03:38.654261 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.654129 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-228571-676b499bc-k76j6" event={"ID":"9c290e9a-af71-48e8-99c6-43056987a08d","Type":"ContainerDied","Data":"cab3d0b6a2e1f5a23715a19f048c1ad303435c95caee94dae757709cd1771975"} Oct 30 08:03:38.654261 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.654159 4951 scope.go:122] "RemoveContainer" containerID="11972b0894266cf0bda6b40d698fcaa63a51bfc34ae04ff127fdc891b8fa6faf" Oct 30 08:03:38.656032 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.656006 4951 generic.go:356] "Generic (PLEG): container finished" podID="399020ed-d53c-41b2-8b3d-298647396c87" containerID="763035ff78ae4ceeec5f0ec9862891754f440b7ea2ce90a6e4e9044b405f7131" exitCode=0 Oct 30 08:03:38.656518 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.656499 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-228571-5995bc74cb-5kvfx" Oct 30 08:03:38.656661 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.656637 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-228571-5995bc74cb-5kvfx" event={"ID":"399020ed-d53c-41b2-8b3d-298647396c87","Type":"ContainerDied","Data":"763035ff78ae4ceeec5f0ec9862891754f440b7ea2ce90a6e4e9044b405f7131"} Oct 30 08:03:38.656743 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.656689 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-228571-5995bc74cb-5kvfx" event={"ID":"399020ed-d53c-41b2-8b3d-298647396c87","Type":"ContainerDied","Data":"dc54c976b1438b3ab3c70c9f9bbeeac57b5608e8e3120546b736c953e7ae1481"} Oct 30 08:03:38.665725 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.665704 4951 scope.go:122] "RemoveContainer" containerID="11972b0894266cf0bda6b40d698fcaa63a51bfc34ae04ff127fdc891b8fa6faf" Oct 30 08:03:38.665987 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:03:38.665970 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11972b0894266cf0bda6b40d698fcaa63a51bfc34ae04ff127fdc891b8fa6faf\": container with ID starting with 11972b0894266cf0bda6b40d698fcaa63a51bfc34ae04ff127fdc891b8fa6faf not found: ID does not exist" containerID="11972b0894266cf0bda6b40d698fcaa63a51bfc34ae04ff127fdc891b8fa6faf" Oct 30 08:03:38.666041 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.665994 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11972b0894266cf0bda6b40d698fcaa63a51bfc34ae04ff127fdc891b8fa6faf"} err="failed to get container status \"11972b0894266cf0bda6b40d698fcaa63a51bfc34ae04ff127fdc891b8fa6faf\": rpc error: code = NotFound desc = could not find container \"11972b0894266cf0bda6b40d698fcaa63a51bfc34ae04ff127fdc891b8fa6faf\": container with ID starting with 11972b0894266cf0bda6b40d698fcaa63a51bfc34ae04ff127fdc891b8fa6faf not found: ID does not exist" Oct 30 08:03:38.666041 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.666012 4951 scope.go:122] "RemoveContainer" containerID="763035ff78ae4ceeec5f0ec9862891754f440b7ea2ce90a6e4e9044b405f7131" Oct 30 08:03:38.677753 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.677731 4951 scope.go:122] "RemoveContainer" containerID="763035ff78ae4ceeec5f0ec9862891754f440b7ea2ce90a6e4e9044b405f7131" Oct 30 08:03:38.678016 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:03:38.678000 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"763035ff78ae4ceeec5f0ec9862891754f440b7ea2ce90a6e4e9044b405f7131\": container with ID starting with 763035ff78ae4ceeec5f0ec9862891754f440b7ea2ce90a6e4e9044b405f7131 not found: ID does not exist" containerID="763035ff78ae4ceeec5f0ec9862891754f440b7ea2ce90a6e4e9044b405f7131" Oct 30 08:03:38.678072 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.678023 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"763035ff78ae4ceeec5f0ec9862891754f440b7ea2ce90a6e4e9044b405f7131"} err="failed to get container status \"763035ff78ae4ceeec5f0ec9862891754f440b7ea2ce90a6e4e9044b405f7131\": rpc error: code = NotFound desc = could not find container \"763035ff78ae4ceeec5f0ec9862891754f440b7ea2ce90a6e4e9044b405f7131\": container with ID starting with 763035ff78ae4ceeec5f0ec9862891754f440b7ea2ce90a6e4e9044b405f7131 not found: ID does not exist" Oct 30 08:03:38.707954 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.707927 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-228571-5995bc74cb-5kvfx"] Oct 30 08:03:38.721530 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.721502 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-228571-5995bc74cb-5kvfx"] Oct 30 08:03:38.753552 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.753519 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-228571-676b499bc-k76j6"] Oct 30 08:03:38.757544 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:38.757511 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-228571-676b499bc-k76j6"] Oct 30 08:03:39.178503 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:39.178416 4951 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Oct 30 08:03:39.178503 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:39.178479 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" podUID="a796d7df-446c-419e-be91-f479e9bfa8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Oct 30 08:03:39.773935 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:39.773899 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="399020ed-d53c-41b2-8b3d-298647396c87" path="/var/lib/kubelet/pods/399020ed-d53c-41b2-8b3d-298647396c87/volumes" Oct 30 08:03:39.774397 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:39.774382 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c290e9a-af71-48e8-99c6-43056987a08d" path="/var/lib/kubelet/pods/9c290e9a-af71-48e8-99c6-43056987a08d/volumes" Oct 30 08:03:40.177959 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:40.177864 4951 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Oct 30 08:03:40.177959 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:40.177929 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" podUID="a796d7df-446c-419e-be91-f479e9bfa8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Oct 30 08:03:41.120912 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:41.120866 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw"] Oct 30 08:03:41.121455 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:41.121135 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw" podUID="ffb358d7-f1b3-4e53-b19e-85031c692e2b" containerName="maas-api" containerID="cri-o://187fb754eecfd756b51be689da6df481cbc4746747d79bcb10115e3cf14a3377" gracePeriod=30 Oct 30 08:03:41.184866 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:41.179034 4951 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Oct 30 08:03:41.184866 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:41.179095 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" podUID="a796d7df-446c-419e-be91-f479e9bfa8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Oct 30 08:03:41.732676 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:41.732345 4951 generic.go:356] "Generic (PLEG): container finished" podID="ffb358d7-f1b3-4e53-b19e-85031c692e2b" containerID="187fb754eecfd756b51be689da6df481cbc4746747d79bcb10115e3cf14a3377" exitCode=0 Oct 30 08:03:41.732676 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:41.732609 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw" event={"ID":"ffb358d7-f1b3-4e53-b19e-85031c692e2b","Type":"ContainerDied","Data":"187fb754eecfd756b51be689da6df481cbc4746747d79bcb10115e3cf14a3377"} Oct 30 08:03:41.812248 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:41.812222 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw" Oct 30 08:03:41.820684 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:41.820658 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffb358d7-f1b3-4e53-b19e-85031c692e2b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffb358d7-f1b3-4e53-b19e-85031c692e2b-metrics-certs\") pod \"ffb358d7-f1b3-4e53-b19e-85031c692e2b\" (UID: \"ffb358d7-f1b3-4e53-b19e-85031c692e2b\") " Oct 30 08:03:41.820835 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:41.820739 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffb358d7-f1b3-4e53-b19e-85031c692e2b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffb358d7-f1b3-4e53-b19e-85031c692e2b-openshift-service-ca\") pod \"ffb358d7-f1b3-4e53-b19e-85031c692e2b\" (UID: \"ffb358d7-f1b3-4e53-b19e-85031c692e2b\") " Oct 30 08:03:41.820835 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:41.820784 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffb358d7-f1b3-4e53-b19e-85031c692e2b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffb358d7-f1b3-4e53-b19e-85031c692e2b-maas-api-tls\") pod \"ffb358d7-f1b3-4e53-b19e-85031c692e2b\" (UID: \"ffb358d7-f1b3-4e53-b19e-85031c692e2b\") " Oct 30 08:03:41.820961 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:41.820859 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffb358d7-f1b3-4e53-b19e-85031c692e2b-kube-api-access-bmccp\" (UniqueName: \"kubernetes.io/projected/ffb358d7-f1b3-4e53-b19e-85031c692e2b-kube-api-access-bmccp\") pod \"ffb358d7-f1b3-4e53-b19e-85031c692e2b\" (UID: \"ffb358d7-f1b3-4e53-b19e-85031c692e2b\") " Oct 30 08:03:41.821250 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:41.821223 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffb358d7-f1b3-4e53-b19e-85031c692e2b-openshift-service-ca" pod "ffb358d7-f1b3-4e53-b19e-85031c692e2b" (UID: "ffb358d7-f1b3-4e53-b19e-85031c692e2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:03:41.851792 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:41.844346 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffb358d7-f1b3-4e53-b19e-85031c692e2b-metrics-certs" pod "ffb358d7-f1b3-4e53-b19e-85031c692e2b" (UID: "ffb358d7-f1b3-4e53-b19e-85031c692e2b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:03:41.851792 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:41.849317 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffb358d7-f1b3-4e53-b19e-85031c692e2b-maas-api-tls" pod "ffb358d7-f1b3-4e53-b19e-85031c692e2b" (UID: "ffb358d7-f1b3-4e53-b19e-85031c692e2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:03:41.851792 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:41.849486 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffb358d7-f1b3-4e53-b19e-85031c692e2b-kube-api-access-bmccp" pod "ffb358d7-f1b3-4e53-b19e-85031c692e2b" (UID: "ffb358d7-f1b3-4e53-b19e-85031c692e2b"). InnerVolumeSpecName "kube-api-access-bmccp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:03:41.926598 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:41.922269 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmccp\" (UniqueName: \"kubernetes.io/projected/ffb358d7-f1b3-4e53-b19e-85031c692e2b-kube-api-access-bmccp\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:41.926598 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:41.922316 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffb358d7-f1b3-4e53-b19e-85031c692e2b-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:41.926598 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:41.922331 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffb358d7-f1b3-4e53-b19e-85031c692e2b-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:41.926598 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:41.922344 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffb358d7-f1b3-4e53-b19e-85031c692e2b-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:42.178511 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:42.178473 4951 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Oct 30 08:03:42.179026 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:42.178533 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" podUID="a796d7df-446c-419e-be91-f479e9bfa8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Oct 30 08:03:42.743549 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:42.743511 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw" event={"ID":"ffb358d7-f1b3-4e53-b19e-85031c692e2b","Type":"ContainerDied","Data":"7c65dabb961feac6d0f5b6081dd74216333cd76faca9cc430f8658920ab5cf6d"} Oct 30 08:03:42.743549 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:42.743529 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw" Oct 30 08:03:42.743855 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:42.743594 4951 scope.go:122] "RemoveContainer" containerID="187fb754eecfd756b51be689da6df481cbc4746747d79bcb10115e3cf14a3377" Oct 30 08:03:42.841689 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:42.841641 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw"] Oct 30 08:03:42.843822 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:42.843793 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bffe43-55d78cd6c9-5bxgw"] Oct 30 08:03:43.209264 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:43.209225 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:03:43.210391 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:43.210366 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:03:43.212285 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:43.212260 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:03:43.776149 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:43.776106 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffb358d7-f1b3-4e53-b19e-85031c692e2b" path="/var/lib/kubelet/pods/ffb358d7-f1b3-4e53-b19e-85031c692e2b/volumes" Oct 30 08:03:49.358618 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:49.358562 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7"] Oct 30 08:03:49.359049 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:49.358832 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" podUID="f52156d3-44fc-4983-984c-8ac36b1cb942" containerName="istio-proxy" containerID="cri-o://221f7866e13f9415e02a182c85468354be59fe340d67e59e135aa0d0746a5417" gracePeriod=30 Oct 30 08:03:51.217219 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:51.217182 4951 patch_prober.go:29] interesting pod/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 08:03:51.217635 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:51.217239 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" podUID="f52156d3-44fc-4983-984c-8ac36b1cb942" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 08:03:53.062721 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:03:53.056774 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 08:03:54.663539 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.663509 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 08:03:54.705721 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.705689 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-workload-certs\") pod \"f52156d3-44fc-4983-984c-8ac36b1cb942\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " Oct 30 08:03:54.705988 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.705750 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-workload-socket\") pod \"f52156d3-44fc-4983-984c-8ac36b1cb942\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " Oct 30 08:03:54.705988 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.705813 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f52156d3-44fc-4983-984c-8ac36b1cb942-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f52156d3-44fc-4983-984c-8ac36b1cb942-istio-podinfo\") pod \"f52156d3-44fc-4983-984c-8ac36b1cb942\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " Oct 30 08:03:54.705988 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.705833 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f52156d3-44fc-4983-984c-8ac36b1cb942-istio-token\" (UniqueName: \"kubernetes.io/projected/f52156d3-44fc-4983-984c-8ac36b1cb942-istio-token\") pod \"f52156d3-44fc-4983-984c-8ac36b1cb942\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " Oct 30 08:03:54.705988 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.705859 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f52156d3-44fc-4983-984c-8ac36b1cb942-kube-api-access-j2rw7\" (UniqueName: \"kubernetes.io/projected/f52156d3-44fc-4983-984c-8ac36b1cb942-kube-api-access-j2rw7\") pod \"f52156d3-44fc-4983-984c-8ac36b1cb942\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " Oct 30 08:03:54.705988 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.705909 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-credential-socket\") pod \"f52156d3-44fc-4983-984c-8ac36b1cb942\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " Oct 30 08:03:54.705988 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.705943 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-istio-envoy\") pod \"f52156d3-44fc-4983-984c-8ac36b1cb942\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " Oct 30 08:03:54.705988 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.705965 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-istio-data\") pod \"f52156d3-44fc-4983-984c-8ac36b1cb942\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " Oct 30 08:03:54.706376 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.706024 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f52156d3-44fc-4983-984c-8ac36b1cb942-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f52156d3-44fc-4983-984c-8ac36b1cb942-istiod-ca-cert\") pod \"f52156d3-44fc-4983-984c-8ac36b1cb942\" (UID: \"f52156d3-44fc-4983-984c-8ac36b1cb942\") " Oct 30 08:03:54.706376 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.706034 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-workload-certs" pod "f52156d3-44fc-4983-984c-8ac36b1cb942" (UID: "f52156d3-44fc-4983-984c-8ac36b1cb942"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:03:54.706376 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.706064 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-workload-socket" pod "f52156d3-44fc-4983-984c-8ac36b1cb942" (UID: "f52156d3-44fc-4983-984c-8ac36b1cb942"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:03:54.706376 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.706246 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-credential-socket" pod "f52156d3-44fc-4983-984c-8ac36b1cb942" (UID: "f52156d3-44fc-4983-984c-8ac36b1cb942"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:03:54.706376 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.706294 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-istio-data" pod "f52156d3-44fc-4983-984c-8ac36b1cb942" (UID: "f52156d3-44fc-4983-984c-8ac36b1cb942"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:03:54.706678 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.706449 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f52156d3-44fc-4983-984c-8ac36b1cb942-istiod-ca-cert" pod "f52156d3-44fc-4983-984c-8ac36b1cb942" (UID: "f52156d3-44fc-4983-984c-8ac36b1cb942"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:03:54.706811 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.706785 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:54.706811 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.706814 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:54.707014 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.706828 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:54.707014 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.706839 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:54.707014 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.706848 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f52156d3-44fc-4983-984c-8ac36b1cb942-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:54.708378 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.708351 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f52156d3-44fc-4983-984c-8ac36b1cb942-kube-api-access-j2rw7" pod "f52156d3-44fc-4983-984c-8ac36b1cb942" (UID: "f52156d3-44fc-4983-984c-8ac36b1cb942"). InnerVolumeSpecName "kube-api-access-j2rw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:03:54.708490 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.708407 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f52156d3-44fc-4983-984c-8ac36b1cb942-istio-token" pod "f52156d3-44fc-4983-984c-8ac36b1cb942" (UID: "f52156d3-44fc-4983-984c-8ac36b1cb942"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:03:54.708636 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.708617 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f52156d3-44fc-4983-984c-8ac36b1cb942-istio-podinfo" pod "f52156d3-44fc-4983-984c-8ac36b1cb942" (UID: "f52156d3-44fc-4983-984c-8ac36b1cb942"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:03:54.708705 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.708671 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-istio-envoy" pod "f52156d3-44fc-4983-984c-8ac36b1cb942" (UID: "f52156d3-44fc-4983-984c-8ac36b1cb942"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:03:54.808996 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.808950 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f52156d3-44fc-4983-984c-8ac36b1cb942-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:54.808996 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.808984 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f52156d3-44fc-4983-984c-8ac36b1cb942-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:54.808996 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.808993 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2rw7\" (UniqueName: \"kubernetes.io/projected/f52156d3-44fc-4983-984c-8ac36b1cb942-kube-api-access-j2rw7\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:54.808996 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.809004 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f52156d3-44fc-4983-984c-8ac36b1cb942-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:03:54.833239 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.833203 4951 generic.go:356] "Generic (PLEG): container finished" podID="f52156d3-44fc-4983-984c-8ac36b1cb942" containerID="221f7866e13f9415e02a182c85468354be59fe340d67e59e135aa0d0746a5417" exitCode=0 Oct 30 08:03:54.833239 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.833260 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" Oct 30 08:03:54.833483 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.833265 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" event={"ID":"f52156d3-44fc-4983-984c-8ac36b1cb942","Type":"ContainerDied","Data":"221f7866e13f9415e02a182c85468354be59fe340d67e59e135aa0d0746a5417"} Oct 30 08:03:54.833483 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.833383 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7" event={"ID":"f52156d3-44fc-4983-984c-8ac36b1cb942","Type":"ContainerDied","Data":"f41a46da4d7b2f8f2fd81f5b38c94ad495bf4a38c7ceeeca4b0af627f61f33d5"} Oct 30 08:03:54.833483 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.833415 4951 scope.go:122] "RemoveContainer" containerID="221f7866e13f9415e02a182c85468354be59fe340d67e59e135aa0d0746a5417" Oct 30 08:03:54.855413 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.855388 4951 scope.go:122] "RemoveContainer" containerID="0d56edeb5190f4570765bd8943b7ca7c92294d37c62e21b5831dcb036331a833" Oct 30 08:03:54.886558 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.886200 4951 scope.go:122] "RemoveContainer" containerID="221f7866e13f9415e02a182c85468354be59fe340d67e59e135aa0d0746a5417" Oct 30 08:03:54.888445 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:03:54.887891 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"221f7866e13f9415e02a182c85468354be59fe340d67e59e135aa0d0746a5417\": container with ID starting with 221f7866e13f9415e02a182c85468354be59fe340d67e59e135aa0d0746a5417 not found: ID does not exist" containerID="221f7866e13f9415e02a182c85468354be59fe340d67e59e135aa0d0746a5417" Oct 30 08:03:54.888445 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.887945 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"221f7866e13f9415e02a182c85468354be59fe340d67e59e135aa0d0746a5417"} err="failed to get container status \"221f7866e13f9415e02a182c85468354be59fe340d67e59e135aa0d0746a5417\": rpc error: code = NotFound desc = could not find container \"221f7866e13f9415e02a182c85468354be59fe340d67e59e135aa0d0746a5417\": container with ID starting with 221f7866e13f9415e02a182c85468354be59fe340d67e59e135aa0d0746a5417 not found: ID does not exist" Oct 30 08:03:54.888445 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.887976 4951 scope.go:122] "RemoveContainer" containerID="0d56edeb5190f4570765bd8943b7ca7c92294d37c62e21b5831dcb036331a833" Oct 30 08:03:54.889016 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:03:54.888980 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d56edeb5190f4570765bd8943b7ca7c92294d37c62e21b5831dcb036331a833\": container with ID starting with 0d56edeb5190f4570765bd8943b7ca7c92294d37c62e21b5831dcb036331a833 not found: ID does not exist" containerID="0d56edeb5190f4570765bd8943b7ca7c92294d37c62e21b5831dcb036331a833" Oct 30 08:03:54.889096 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.889028 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d56edeb5190f4570765bd8943b7ca7c92294d37c62e21b5831dcb036331a833"} err="failed to get container status \"0d56edeb5190f4570765bd8943b7ca7c92294d37c62e21b5831dcb036331a833\": rpc error: code = NotFound desc = could not find container \"0d56edeb5190f4570765bd8943b7ca7c92294d37c62e21b5831dcb036331a833\": container with ID starting with 0d56edeb5190f4570765bd8943b7ca7c92294d37c62e21b5831dcb036331a833 not found: ID does not exist" Oct 30 08:03:54.912254 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.912218 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7"] Oct 30 08:03:54.918019 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:54.917991 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-bffe43-openshift-default-75549fcb9-vglw7"] Oct 30 08:03:55.774731 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:03:55.774700 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f52156d3-44fc-4983-984c-8ac36b1cb942" path="/var/lib/kubelet/pods/f52156d3-44fc-4983-984c-8ac36b1cb942/volumes" Oct 30 08:04:02.893010 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:02.892965 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd"] Oct 30 08:04:02.893614 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:02.893250 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd" podUID="0628b544-4ce0-4902-a5e8-5fef9194d1a2" containerName="maas-api" containerID="cri-o://1345c4ee041facb5882df14d6b7bd55e683ae3e959ae8ee66f184b8daaf4a953" gracePeriod=30 Oct 30 08:04:03.261490 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:03.255868 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd" Oct 30 08:04:03.300593 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:03.296807 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0628b544-4ce0-4902-a5e8-5fef9194d1a2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0628b544-4ce0-4902-a5e8-5fef9194d1a2-metrics-certs\") pod \"0628b544-4ce0-4902-a5e8-5fef9194d1a2\" (UID: \"0628b544-4ce0-4902-a5e8-5fef9194d1a2\") " Oct 30 08:04:03.300593 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:03.296878 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0628b544-4ce0-4902-a5e8-5fef9194d1a2-kube-api-access-4vx8t\" (UniqueName: \"kubernetes.io/projected/0628b544-4ce0-4902-a5e8-5fef9194d1a2-kube-api-access-4vx8t\") pod \"0628b544-4ce0-4902-a5e8-5fef9194d1a2\" (UID: \"0628b544-4ce0-4902-a5e8-5fef9194d1a2\") " Oct 30 08:04:03.300593 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:03.297107 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0628b544-4ce0-4902-a5e8-5fef9194d1a2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0628b544-4ce0-4902-a5e8-5fef9194d1a2-maas-api-tls\") pod \"0628b544-4ce0-4902-a5e8-5fef9194d1a2\" (UID: \"0628b544-4ce0-4902-a5e8-5fef9194d1a2\") " Oct 30 08:04:03.300593 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:03.297403 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0628b544-4ce0-4902-a5e8-5fef9194d1a2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0628b544-4ce0-4902-a5e8-5fef9194d1a2-openshift-service-ca\") pod \"0628b544-4ce0-4902-a5e8-5fef9194d1a2\" (UID: \"0628b544-4ce0-4902-a5e8-5fef9194d1a2\") " Oct 30 08:04:03.300593 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:03.298933 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0628b544-4ce0-4902-a5e8-5fef9194d1a2-openshift-service-ca" pod "0628b544-4ce0-4902-a5e8-5fef9194d1a2" (UID: "0628b544-4ce0-4902-a5e8-5fef9194d1a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:04:03.320591 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:03.313422 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0628b544-4ce0-4902-a5e8-5fef9194d1a2-kube-api-access-4vx8t" pod "0628b544-4ce0-4902-a5e8-5fef9194d1a2" (UID: "0628b544-4ce0-4902-a5e8-5fef9194d1a2"). InnerVolumeSpecName "kube-api-access-4vx8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:04:03.336240 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:03.331944 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0628b544-4ce0-4902-a5e8-5fef9194d1a2-metrics-certs" pod "0628b544-4ce0-4902-a5e8-5fef9194d1a2" (UID: "0628b544-4ce0-4902-a5e8-5fef9194d1a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:04:03.353112 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:03.351707 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0628b544-4ce0-4902-a5e8-5fef9194d1a2-maas-api-tls" pod "0628b544-4ce0-4902-a5e8-5fef9194d1a2" (UID: "0628b544-4ce0-4902-a5e8-5fef9194d1a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:04:03.406612 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:03.405958 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0628b544-4ce0-4902-a5e8-5fef9194d1a2-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:03.406612 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:03.405991 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0628b544-4ce0-4902-a5e8-5fef9194d1a2-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:03.406612 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:03.406001 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0628b544-4ce0-4902-a5e8-5fef9194d1a2-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:03.406612 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:03.406010 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vx8t\" (UniqueName: \"kubernetes.io/projected/0628b544-4ce0-4902-a5e8-5fef9194d1a2-kube-api-access-4vx8t\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:03.931564 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:03.931528 4951 generic.go:356] "Generic (PLEG): container finished" podID="0628b544-4ce0-4902-a5e8-5fef9194d1a2" containerID="1345c4ee041facb5882df14d6b7bd55e683ae3e959ae8ee66f184b8daaf4a953" exitCode=0 Oct 30 08:04:03.931564 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:03.931589 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd" event={"ID":"0628b544-4ce0-4902-a5e8-5fef9194d1a2","Type":"ContainerDied","Data":"1345c4ee041facb5882df14d6b7bd55e683ae3e959ae8ee66f184b8daaf4a953"} Oct 30 08:04:03.932088 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:03.931607 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd" Oct 30 08:04:03.932088 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:03.931642 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd" event={"ID":"0628b544-4ce0-4902-a5e8-5fef9194d1a2","Type":"ContainerDied","Data":"48d5bb2313c13c9dd4c8581bbf8ad51743d69d786827cb7ca86d2eb707def89b"} Oct 30 08:04:03.932088 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:03.931670 4951 scope.go:122] "RemoveContainer" containerID="1345c4ee041facb5882df14d6b7bd55e683ae3e959ae8ee66f184b8daaf4a953" Oct 30 08:04:03.944231 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:03.944206 4951 scope.go:122] "RemoveContainer" containerID="1345c4ee041facb5882df14d6b7bd55e683ae3e959ae8ee66f184b8daaf4a953" Oct 30 08:04:03.944726 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:04:03.944687 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1345c4ee041facb5882df14d6b7bd55e683ae3e959ae8ee66f184b8daaf4a953\": container with ID starting with 1345c4ee041facb5882df14d6b7bd55e683ae3e959ae8ee66f184b8daaf4a953 not found: ID does not exist" containerID="1345c4ee041facb5882df14d6b7bd55e683ae3e959ae8ee66f184b8daaf4a953" Oct 30 08:04:03.944818 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:03.944739 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1345c4ee041facb5882df14d6b7bd55e683ae3e959ae8ee66f184b8daaf4a953"} err="failed to get container status \"1345c4ee041facb5882df14d6b7bd55e683ae3e959ae8ee66f184b8daaf4a953\": rpc error: code = NotFound desc = could not find container \"1345c4ee041facb5882df14d6b7bd55e683ae3e959ae8ee66f184b8daaf4a953\": container with ID starting with 1345c4ee041facb5882df14d6b7bd55e683ae3e959ae8ee66f184b8daaf4a953 not found: ID does not exist" Oct 30 08:04:03.987035 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:03.986997 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd"] Oct 30 08:04:03.999564 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:03.999535 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-228571-69df78c74b-wzctd"] Oct 30 08:04:05.774385 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:05.774347 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0628b544-4ce0-4902-a5e8-5fef9194d1a2" path="/var/lib/kubelet/pods/0628b544-4ce0-4902-a5e8-5fef9194d1a2/volumes" Oct 30 08:04:11.411613 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:11.405663 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt"] Oct 30 08:04:11.411613 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:11.405946 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" podUID="bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4" containerName="istio-proxy" containerID="cri-o://8e3aeacad8d45b21779e48bae2b6239d55a009bbd360118015b2d907cee78997" gracePeriod=30 Oct 30 08:04:12.002501 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:12.002457 4951 patch_prober.go:29] interesting pod/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 08:04:12.002718 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:12.002537 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" podUID="bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 08:04:16.699680 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:16.699649 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 08:04:16.822371 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:16.822327 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-credential-socket\") pod \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " Oct 30 08:04:16.822633 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:16.822401 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istiod-ca-cert\") pod \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " Oct 30 08:04:16.822633 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:16.822432 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istio-podinfo\") pod \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " Oct 30 08:04:16.822633 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:16.822549 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istio-token\" (UniqueName: \"kubernetes.io/projected/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istio-token\") pod \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " Oct 30 08:04:16.822855 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:16.822645 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istio-data\") pod \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " Oct 30 08:04:16.822855 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:16.822678 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-workload-certs\") pod \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " Oct 30 08:04:16.822855 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:16.822706 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-workload-socket\") pod \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " Oct 30 08:04:16.822855 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:16.822720 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-credential-socket" pod "bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4" (UID: "bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:04:16.822855 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:16.822762 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istio-envoy\") pod \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " Oct 30 08:04:16.822855 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:16.822821 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-kube-api-access-hcpw9\" (UniqueName: \"kubernetes.io/projected/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-kube-api-access-hcpw9\") pod \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\" (UID: \"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4\") " Oct 30 08:04:16.823263 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:16.823240 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-workload-socket" pod "bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4" (UID: "bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:04:16.823328 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:16.823281 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-workload-certs" pod "bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4" (UID: "bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:04:16.823524 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:16.823499 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:16.823524 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:16.823525 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:16.824104 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:16.823540 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:16.824104 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:16.823539 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istio-data" pod "bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4" (UID: "bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:04:16.824104 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:16.823559 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istiod-ca-cert" pod "bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4" (UID: "bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:04:16.825459 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:16.825423 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istio-token" pod "bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4" (UID: "bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:04:16.825591 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:16.825469 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istio-podinfo" pod "bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4" (UID: "bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:04:16.825591 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:16.825539 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-kube-api-access-hcpw9" pod "bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4" (UID: "bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4"). InnerVolumeSpecName "kube-api-access-hcpw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:04:16.825714 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:16.825645 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istio-envoy" pod "bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4" (UID: "bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:04:16.899557 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:16.899526 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv"] Oct 30 08:04:16.915144 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:16.915117 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk"] Oct 30 08:04:16.925636 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:16.925544 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:16.925636 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:16.925588 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:16.925636 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:16.925602 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:16.925870 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:16.925645 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:16.925870 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:16.925659 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:16.925870 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:16.925672 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcpw9\" (UniqueName: \"kubernetes.io/projected/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4-kube-api-access-hcpw9\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:16.928310 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:16.928289 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct"] Oct 30 08:04:17.027561 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:17.027530 4951 generic.go:356] "Generic (PLEG): container finished" podID="bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4" containerID="8e3aeacad8d45b21779e48bae2b6239d55a009bbd360118015b2d907cee78997" exitCode=0 Oct 30 08:04:17.027751 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:17.027620 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" event={"ID":"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4","Type":"ContainerDied","Data":"8e3aeacad8d45b21779e48bae2b6239d55a009bbd360118015b2d907cee78997"} Oct 30 08:04:17.027751 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:17.027678 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" event={"ID":"bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4","Type":"ContainerDied","Data":"9dfc129ab83cfe45f21f5709f2e75324802b3fdad4020e112291f2473c5072cf"} Oct 30 08:04:17.027751 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:17.027626 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt" Oct 30 08:04:17.027751 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:17.027702 4951 scope.go:122] "RemoveContainer" containerID="8e3aeacad8d45b21779e48bae2b6239d55a009bbd360118015b2d907cee78997" Oct 30 08:04:17.039634 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:17.039564 4951 scope.go:122] "RemoveContainer" containerID="91a57d17aefb7d1502c0f545ac473c165cd5beada4caf3c52a66d044476b5836" Oct 30 08:04:17.049626 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:17.049606 4951 scope.go:122] "RemoveContainer" containerID="8e3aeacad8d45b21779e48bae2b6239d55a009bbd360118015b2d907cee78997" Oct 30 08:04:17.049972 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:04:17.049952 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e3aeacad8d45b21779e48bae2b6239d55a009bbd360118015b2d907cee78997\": container with ID starting with 8e3aeacad8d45b21779e48bae2b6239d55a009bbd360118015b2d907cee78997 not found: ID does not exist" containerID="8e3aeacad8d45b21779e48bae2b6239d55a009bbd360118015b2d907cee78997" Oct 30 08:04:17.050040 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:17.049985 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e3aeacad8d45b21779e48bae2b6239d55a009bbd360118015b2d907cee78997"} err="failed to get container status \"8e3aeacad8d45b21779e48bae2b6239d55a009bbd360118015b2d907cee78997\": rpc error: code = NotFound desc = could not find container \"8e3aeacad8d45b21779e48bae2b6239d55a009bbd360118015b2d907cee78997\": container with ID starting with 8e3aeacad8d45b21779e48bae2b6239d55a009bbd360118015b2d907cee78997 not found: ID does not exist" Oct 30 08:04:17.050040 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:17.050005 4951 scope.go:122] "RemoveContainer" containerID="91a57d17aefb7d1502c0f545ac473c165cd5beada4caf3c52a66d044476b5836" Oct 30 08:04:17.050282 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:04:17.050263 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91a57d17aefb7d1502c0f545ac473c165cd5beada4caf3c52a66d044476b5836\": container with ID starting with 91a57d17aefb7d1502c0f545ac473c165cd5beada4caf3c52a66d044476b5836 not found: ID does not exist" containerID="91a57d17aefb7d1502c0f545ac473c165cd5beada4caf3c52a66d044476b5836" Oct 30 08:04:17.050339 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:17.050289 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91a57d17aefb7d1502c0f545ac473c165cd5beada4caf3c52a66d044476b5836"} err="failed to get container status \"91a57d17aefb7d1502c0f545ac473c165cd5beada4caf3c52a66d044476b5836\": rpc error: code = NotFound desc = could not find container \"91a57d17aefb7d1502c0f545ac473c165cd5beada4caf3c52a66d044476b5836\": container with ID starting with 91a57d17aefb7d1502c0f545ac473c165cd5beada4caf3c52a66d044476b5836 not found: ID does not exist" Oct 30 08:04:17.087695 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:17.087658 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt"] Oct 30 08:04:17.092950 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:17.092920 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-228571-openshift-default-76dd4777c9-ct9qt"] Oct 30 08:04:17.773000 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:17.772964 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4" path="/var/lib/kubelet/pods/bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4/volumes" Oct 30 08:04:22.736200 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:22.736158 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b72d79-7fd54fd9d4-hfflc"] Oct 30 08:04:22.772049 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:22.772019 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b72d79-77d45dc4f5-nzd72"] Oct 30 08:04:23.037719 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:04:23.037649 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 08:04:26.178232 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:26.177500 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx"] Oct 30 08:04:26.178232 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:26.177850 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx" podUID="f014cb28-67ea-4356-a8a2-846720aaaa6d" containerName="maas-api" containerID="cri-o://3b2706b6b07c39a6ada37aebaf8bbc88419f3670b74c73863c1e9bac999ab052" gracePeriod=30 Oct 30 08:04:26.532464 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:26.532431 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx" Oct 30 08:04:26.567295 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:26.567249 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f014cb28-67ea-4356-a8a2-846720aaaa6d-kube-api-access-hknrb\" (UniqueName: \"kubernetes.io/projected/f014cb28-67ea-4356-a8a2-846720aaaa6d-kube-api-access-hknrb\") pod \"f014cb28-67ea-4356-a8a2-846720aaaa6d\" (UID: \"f014cb28-67ea-4356-a8a2-846720aaaa6d\") " Oct 30 08:04:26.567492 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:26.567341 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f014cb28-67ea-4356-a8a2-846720aaaa6d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f014cb28-67ea-4356-a8a2-846720aaaa6d-maas-api-tls\") pod \"f014cb28-67ea-4356-a8a2-846720aaaa6d\" (UID: \"f014cb28-67ea-4356-a8a2-846720aaaa6d\") " Oct 30 08:04:26.567492 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:26.567456 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f014cb28-67ea-4356-a8a2-846720aaaa6d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f014cb28-67ea-4356-a8a2-846720aaaa6d-openshift-service-ca\") pod \"f014cb28-67ea-4356-a8a2-846720aaaa6d\" (UID: \"f014cb28-67ea-4356-a8a2-846720aaaa6d\") " Oct 30 08:04:26.567649 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:26.567523 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f014cb28-67ea-4356-a8a2-846720aaaa6d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f014cb28-67ea-4356-a8a2-846720aaaa6d-metrics-certs\") pod \"f014cb28-67ea-4356-a8a2-846720aaaa6d\" (UID: \"f014cb28-67ea-4356-a8a2-846720aaaa6d\") " Oct 30 08:04:26.567898 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:26.567863 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f014cb28-67ea-4356-a8a2-846720aaaa6d-openshift-service-ca" pod "f014cb28-67ea-4356-a8a2-846720aaaa6d" (UID: "f014cb28-67ea-4356-a8a2-846720aaaa6d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:04:26.568327 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:26.568302 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f014cb28-67ea-4356-a8a2-846720aaaa6d-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:26.569932 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:26.569903 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f014cb28-67ea-4356-a8a2-846720aaaa6d-kube-api-access-hknrb" pod "f014cb28-67ea-4356-a8a2-846720aaaa6d" (UID: "f014cb28-67ea-4356-a8a2-846720aaaa6d"). InnerVolumeSpecName "kube-api-access-hknrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:04:26.570169 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:26.570144 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f014cb28-67ea-4356-a8a2-846720aaaa6d-maas-api-tls" pod "f014cb28-67ea-4356-a8a2-846720aaaa6d" (UID: "f014cb28-67ea-4356-a8a2-846720aaaa6d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:04:26.570239 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:26.570198 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f014cb28-67ea-4356-a8a2-846720aaaa6d-metrics-certs" pod "f014cb28-67ea-4356-a8a2-846720aaaa6d" (UID: "f014cb28-67ea-4356-a8a2-846720aaaa6d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:04:26.678694 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:26.669997 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f014cb28-67ea-4356-a8a2-846720aaaa6d-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:26.678694 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:26.670033 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f014cb28-67ea-4356-a8a2-846720aaaa6d-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:26.678694 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:26.670048 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hknrb\" (UniqueName: \"kubernetes.io/projected/f014cb28-67ea-4356-a8a2-846720aaaa6d-kube-api-access-hknrb\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:27.123273 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:27.123237 4951 generic.go:356] "Generic (PLEG): container finished" podID="f014cb28-67ea-4356-a8a2-846720aaaa6d" containerID="3b2706b6b07c39a6ada37aebaf8bbc88419f3670b74c73863c1e9bac999ab052" exitCode=0 Oct 30 08:04:27.123461 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:27.123290 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx" Oct 30 08:04:27.123461 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:27.123309 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx" event={"ID":"f014cb28-67ea-4356-a8a2-846720aaaa6d","Type":"ContainerDied","Data":"3b2706b6b07c39a6ada37aebaf8bbc88419f3670b74c73863c1e9bac999ab052"} Oct 30 08:04:27.123461 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:27.123362 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx" event={"ID":"f014cb28-67ea-4356-a8a2-846720aaaa6d","Type":"ContainerDied","Data":"118c37b0e2a5b0aa59e7f1d09c41188bf276d4b7c08c1ab2e20d97e993cef54e"} Oct 30 08:04:27.123461 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:27.123393 4951 scope.go:122] "RemoveContainer" containerID="3b2706b6b07c39a6ada37aebaf8bbc88419f3670b74c73863c1e9bac999ab052" Oct 30 08:04:27.136425 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:27.136402 4951 scope.go:122] "RemoveContainer" containerID="3b2706b6b07c39a6ada37aebaf8bbc88419f3670b74c73863c1e9bac999ab052" Oct 30 08:04:27.136821 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:04:27.136798 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b2706b6b07c39a6ada37aebaf8bbc88419f3670b74c73863c1e9bac999ab052\": container with ID starting with 3b2706b6b07c39a6ada37aebaf8bbc88419f3670b74c73863c1e9bac999ab052 not found: ID does not exist" containerID="3b2706b6b07c39a6ada37aebaf8bbc88419f3670b74c73863c1e9bac999ab052" Oct 30 08:04:27.136894 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:27.136836 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b2706b6b07c39a6ada37aebaf8bbc88419f3670b74c73863c1e9bac999ab052"} err="failed to get container status \"3b2706b6b07c39a6ada37aebaf8bbc88419f3670b74c73863c1e9bac999ab052\": rpc error: code = NotFound desc = could not find container \"3b2706b6b07c39a6ada37aebaf8bbc88419f3670b74c73863c1e9bac999ab052\": container with ID starting with 3b2706b6b07c39a6ada37aebaf8bbc88419f3670b74c73863c1e9bac999ab052 not found: ID does not exist" Oct 30 08:04:27.179107 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:27.179077 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx"] Oct 30 08:04:27.191474 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:27.191447 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2761aa-7bd47dc688-ptlkx"] Oct 30 08:04:27.737607 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:27.737549 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-b72d79-7fd54fd9d4-hfflc" podUID="cd68fbab-46a8-4bec-8c75-e6d3e07bf691" containerName="payload-processing" containerID="cri-o://079f1d88b4a22bca2ddc4439ade09f3c4b43da61dfe6ae7c361394c2e20dbff3" gracePeriod=25 Oct 30 08:04:27.774135 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:27.774098 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f014cb28-67ea-4356-a8a2-846720aaaa6d" path="/var/lib/kubelet/pods/f014cb28-67ea-4356-a8a2-846720aaaa6d/volumes" Oct 30 08:04:28.026933 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:28.026907 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b72d79-7fd54fd9d4-hfflc" Oct 30 08:04:28.082911 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:28.082265 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b72d79-77d45dc4f5-nzd72" podUID="50da1f49-e013-4e3e-a139-99d4ae6d668f" containerName="payload-pre-processing" containerID="cri-o://5183599832cb3ea3108cba2db7030105f973ac4f09a02934fbc8711daa3c3b17" gracePeriod=25 Oct 30 08:04:28.087403 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:28.087372 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd68fbab-46a8-4bec-8c75-e6d3e07bf691-kube-api-access-rvrb2\" (UniqueName: \"kubernetes.io/projected/cd68fbab-46a8-4bec-8c75-e6d3e07bf691-kube-api-access-rvrb2\") pod \"cd68fbab-46a8-4bec-8c75-e6d3e07bf691\" (UID: \"cd68fbab-46a8-4bec-8c75-e6d3e07bf691\") " Oct 30 08:04:28.087615 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:28.087558 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd68fbab-46a8-4bec-8c75-e6d3e07bf691-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd68fbab-46a8-4bec-8c75-e6d3e07bf691-plugins-config-volume\") pod \"cd68fbab-46a8-4bec-8c75-e6d3e07bf691\" (UID: \"cd68fbab-46a8-4bec-8c75-e6d3e07bf691\") " Oct 30 08:04:28.088055 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:28.088019 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd68fbab-46a8-4bec-8c75-e6d3e07bf691-plugins-config-volume" pod "cd68fbab-46a8-4bec-8c75-e6d3e07bf691" (UID: "cd68fbab-46a8-4bec-8c75-e6d3e07bf691"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:04:28.088483 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:28.088458 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd68fbab-46a8-4bec-8c75-e6d3e07bf691-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:28.090875 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:28.090847 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd68fbab-46a8-4bec-8c75-e6d3e07bf691-kube-api-access-rvrb2" pod "cd68fbab-46a8-4bec-8c75-e6d3e07bf691" (UID: "cd68fbab-46a8-4bec-8c75-e6d3e07bf691"). InnerVolumeSpecName "kube-api-access-rvrb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:04:28.132536 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:28.132501 4951 generic.go:356] "Generic (PLEG): container finished" podID="cd68fbab-46a8-4bec-8c75-e6d3e07bf691" containerID="079f1d88b4a22bca2ddc4439ade09f3c4b43da61dfe6ae7c361394c2e20dbff3" exitCode=0 Oct 30 08:04:28.132712 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:28.132558 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b72d79-7fd54fd9d4-hfflc" Oct 30 08:04:28.132712 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:28.132618 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b72d79-7fd54fd9d4-hfflc" event={"ID":"cd68fbab-46a8-4bec-8c75-e6d3e07bf691","Type":"ContainerDied","Data":"079f1d88b4a22bca2ddc4439ade09f3c4b43da61dfe6ae7c361394c2e20dbff3"} Oct 30 08:04:28.132712 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:28.132703 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b72d79-7fd54fd9d4-hfflc" event={"ID":"cd68fbab-46a8-4bec-8c75-e6d3e07bf691","Type":"ContainerDied","Data":"a1d14e32c3fbdaa56ea32cb60f1270609e14b5172e1ced9ee578e7c65fa79a62"} Oct 30 08:04:28.132890 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:28.132735 4951 scope.go:122] "RemoveContainer" containerID="079f1d88b4a22bca2ddc4439ade09f3c4b43da61dfe6ae7c361394c2e20dbff3" Oct 30 08:04:28.190091 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:28.190051 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvrb2\" (UniqueName: \"kubernetes.io/projected/cd68fbab-46a8-4bec-8c75-e6d3e07bf691-kube-api-access-rvrb2\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:28.196518 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:28.196486 4951 scope.go:122] "RemoveContainer" containerID="079f1d88b4a22bca2ddc4439ade09f3c4b43da61dfe6ae7c361394c2e20dbff3" Oct 30 08:04:28.196941 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:04:28.196917 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"079f1d88b4a22bca2ddc4439ade09f3c4b43da61dfe6ae7c361394c2e20dbff3\": container with ID starting with 079f1d88b4a22bca2ddc4439ade09f3c4b43da61dfe6ae7c361394c2e20dbff3 not found: ID does not exist" containerID="079f1d88b4a22bca2ddc4439ade09f3c4b43da61dfe6ae7c361394c2e20dbff3" Oct 30 08:04:28.197026 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:28.196949 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"079f1d88b4a22bca2ddc4439ade09f3c4b43da61dfe6ae7c361394c2e20dbff3"} err="failed to get container status \"079f1d88b4a22bca2ddc4439ade09f3c4b43da61dfe6ae7c361394c2e20dbff3\": rpc error: code = NotFound desc = could not find container \"079f1d88b4a22bca2ddc4439ade09f3c4b43da61dfe6ae7c361394c2e20dbff3\": container with ID starting with 079f1d88b4a22bca2ddc4439ade09f3c4b43da61dfe6ae7c361394c2e20dbff3 not found: ID does not exist" Oct 30 08:04:28.242301 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:28.242210 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b72d79-7fd54fd9d4-hfflc"] Oct 30 08:04:28.248414 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:28.248378 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b72d79-7fd54fd9d4-hfflc"] Oct 30 08:04:28.344327 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:28.344303 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b72d79-77d45dc4f5-nzd72" Oct 30 08:04:28.393268 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:28.393237 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50da1f49-e013-4e3e-a139-99d4ae6d668f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50da1f49-e013-4e3e-a139-99d4ae6d668f-plugins-config-volume\") pod \"50da1f49-e013-4e3e-a139-99d4ae6d668f\" (UID: \"50da1f49-e013-4e3e-a139-99d4ae6d668f\") " Oct 30 08:04:28.393501 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:28.393482 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50da1f49-e013-4e3e-a139-99d4ae6d668f-kube-api-access-djnnw\" (UniqueName: \"kubernetes.io/projected/50da1f49-e013-4e3e-a139-99d4ae6d668f-kube-api-access-djnnw\") pod \"50da1f49-e013-4e3e-a139-99d4ae6d668f\" (UID: \"50da1f49-e013-4e3e-a139-99d4ae6d668f\") " Oct 30 08:04:28.393710 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:28.393686 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50da1f49-e013-4e3e-a139-99d4ae6d668f-plugins-config-volume" pod "50da1f49-e013-4e3e-a139-99d4ae6d668f" (UID: "50da1f49-e013-4e3e-a139-99d4ae6d668f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:04:28.393982 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:28.393962 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50da1f49-e013-4e3e-a139-99d4ae6d668f-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:28.395849 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:28.395826 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50da1f49-e013-4e3e-a139-99d4ae6d668f-kube-api-access-djnnw" pod "50da1f49-e013-4e3e-a139-99d4ae6d668f" (UID: "50da1f49-e013-4e3e-a139-99d4ae6d668f"). InnerVolumeSpecName "kube-api-access-djnnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:04:28.495716 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:28.495644 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djnnw\" (UniqueName: \"kubernetes.io/projected/50da1f49-e013-4e3e-a139-99d4ae6d668f-kube-api-access-djnnw\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:29.144746 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:29.144662 4951 generic.go:356] "Generic (PLEG): container finished" podID="50da1f49-e013-4e3e-a139-99d4ae6d668f" containerID="5183599832cb3ea3108cba2db7030105f973ac4f09a02934fbc8711daa3c3b17" exitCode=0 Oct 30 08:04:29.144917 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:29.144792 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b72d79-77d45dc4f5-nzd72" event={"ID":"50da1f49-e013-4e3e-a139-99d4ae6d668f","Type":"ContainerDied","Data":"5183599832cb3ea3108cba2db7030105f973ac4f09a02934fbc8711daa3c3b17"} Oct 30 08:04:29.144917 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:29.144850 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b72d79-77d45dc4f5-nzd72" event={"ID":"50da1f49-e013-4e3e-a139-99d4ae6d668f","Type":"ContainerDied","Data":"2c0464fabddd5c33b0e24636830fccff5b6acddbf2d105ce70b5e1ce07bb9291"} Oct 30 08:04:29.144917 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:29.144865 4951 scope.go:122] "RemoveContainer" containerID="5183599832cb3ea3108cba2db7030105f973ac4f09a02934fbc8711daa3c3b17" Oct 30 08:04:29.145141 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:29.144849 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b72d79-77d45dc4f5-nzd72" Oct 30 08:04:29.157034 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:29.157014 4951 scope.go:122] "RemoveContainer" containerID="5183599832cb3ea3108cba2db7030105f973ac4f09a02934fbc8711daa3c3b17" Oct 30 08:04:29.157313 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:04:29.157294 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5183599832cb3ea3108cba2db7030105f973ac4f09a02934fbc8711daa3c3b17\": container with ID starting with 5183599832cb3ea3108cba2db7030105f973ac4f09a02934fbc8711daa3c3b17 not found: ID does not exist" containerID="5183599832cb3ea3108cba2db7030105f973ac4f09a02934fbc8711daa3c3b17" Oct 30 08:04:29.157393 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:29.157320 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5183599832cb3ea3108cba2db7030105f973ac4f09a02934fbc8711daa3c3b17"} err="failed to get container status \"5183599832cb3ea3108cba2db7030105f973ac4f09a02934fbc8711daa3c3b17\": rpc error: code = NotFound desc = could not find container \"5183599832cb3ea3108cba2db7030105f973ac4f09a02934fbc8711daa3c3b17\": container with ID starting with 5183599832cb3ea3108cba2db7030105f973ac4f09a02934fbc8711daa3c3b17 not found: ID does not exist" Oct 30 08:04:29.210506 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:29.210473 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b72d79-77d45dc4f5-nzd72"] Oct 30 08:04:29.222285 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:29.222260 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b72d79-77d45dc4f5-nzd72"] Oct 30 08:04:29.777939 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:29.777892 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50da1f49-e013-4e3e-a139-99d4ae6d668f" path="/var/lib/kubelet/pods/50da1f49-e013-4e3e-a139-99d4ae6d668f/volumes" Oct 30 08:04:29.778610 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:29.778566 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd68fbab-46a8-4bec-8c75-e6d3e07bf691" path="/var/lib/kubelet/pods/cd68fbab-46a8-4bec-8c75-e6d3e07bf691/volumes" Oct 30 08:04:31.921325 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:31.921288 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" containerID="cri-o://449c36e67c4891a77e26ef1ed03c3ca0eebfd10ef835103293fc97656d434e5e" gracePeriod=45 Oct 30 08:04:31.935811 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:31.935780 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" containerID="cri-o://6c4fa67a100d7e4764ab97b771c3999de42eac25335905cf81ca9a426c19b8fe" gracePeriod=45 Oct 30 08:04:31.948934 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:31.948902 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" containerID="cri-o://7f032fb28d453ab0b355d92f034f3bb800e34c6034836b17077a71166a4d56ef" gracePeriod=45 Oct 30 08:04:31.991720 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:31.991685 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": dial tcp 10.132.0.69:8000: connect: connection refused" Oct 30 08:04:32.176791 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.176759 4951 generic.go:356] "Generic (PLEG): container finished" podID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerID="6c4fa67a100d7e4764ab97b771c3999de42eac25335905cf81ca9a426c19b8fe" exitCode=0 Oct 30 08:04:32.176957 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.176851 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" event={"ID":"d680a08d-af67-455d-87bc-9fa6a26ad37a","Type":"ContainerDied","Data":"6c4fa67a100d7e4764ab97b771c3999de42eac25335905cf81ca9a426c19b8fe"} Oct 30 08:04:32.179065 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.178834 4951 generic.go:356] "Generic (PLEG): container finished" podID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerID="449c36e67c4891a77e26ef1ed03c3ca0eebfd10ef835103293fc97656d434e5e" exitCode=0 Oct 30 08:04:32.179065 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.179008 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" event={"ID":"61d8d83a-32ea-43b0-9aa7-b70bff2217b8","Type":"ContainerDied","Data":"449c36e67c4891a77e26ef1ed03c3ca0eebfd10ef835103293fc97656d434e5e"} Oct 30 08:04:32.181223 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.181192 4951 generic.go:356] "Generic (PLEG): container finished" podID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerID="7f032fb28d453ab0b355d92f034f3bb800e34c6034836b17077a71166a4d56ef" exitCode=0 Oct 30 08:04:32.181328 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.181314 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" event={"ID":"fab18d8f-7bf2-44eb-a099-fd2a60990d0a","Type":"ContainerDied","Data":"7f032fb28d453ab0b355d92f034f3bb800e34c6034836b17077a71166a4d56ef"} Oct 30 08:04:32.284339 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.283815 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 08:04:32.286500 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.286428 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 08:04:32.296928 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.296903 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 08:04:32.382044 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.382017 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-tmp-dir\") pod \"fab18d8f-7bf2-44eb-a099-fd2a60990d0a\" (UID: \"fab18d8f-7bf2-44eb-a099-fd2a60990d0a\") " Oct 30 08:04:32.382332 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.382305 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d680a08d-af67-455d-87bc-9fa6a26ad37a-tls-certs\" (UniqueName: \"kubernetes.io/secret/d680a08d-af67-455d-87bc-9fa6a26ad37a-tls-certs\") pod \"d680a08d-af67-455d-87bc-9fa6a26ad37a\" (UID: \"d680a08d-af67-455d-87bc-9fa6a26ad37a\") " Oct 30 08:04:32.382457 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.382348 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-dshm\") pod \"61d8d83a-32ea-43b0-9aa7-b70bff2217b8\" (UID: \"61d8d83a-32ea-43b0-9aa7-b70bff2217b8\") " Oct 30 08:04:32.382457 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.382305 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-tmp-dir" pod "fab18d8f-7bf2-44eb-a099-fd2a60990d0a" (UID: "fab18d8f-7bf2-44eb-a099-fd2a60990d0a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:04:32.382457 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.382422 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-dshm\") pod \"fab18d8f-7bf2-44eb-a099-fd2a60990d0a\" (UID: \"fab18d8f-7bf2-44eb-a099-fd2a60990d0a\") " Oct 30 08:04:32.382457 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.382452 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-model-cache\") pod \"61d8d83a-32ea-43b0-9aa7-b70bff2217b8\" (UID: \"61d8d83a-32ea-43b0-9aa7-b70bff2217b8\") " Oct 30 08:04:32.382759 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.382527 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d680a08d-af67-455d-87bc-9fa6a26ad37a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d680a08d-af67-455d-87bc-9fa6a26ad37a-tmp-dir\") pod \"d680a08d-af67-455d-87bc-9fa6a26ad37a\" (UID: \"d680a08d-af67-455d-87bc-9fa6a26ad37a\") " Oct 30 08:04:32.382759 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.382555 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-kube-api-access-n89sr\" (UniqueName: \"kubernetes.io/projected/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-kube-api-access-n89sr\") pod \"61d8d83a-32ea-43b0-9aa7-b70bff2217b8\" (UID: \"61d8d83a-32ea-43b0-9aa7-b70bff2217b8\") " Oct 30 08:04:32.382759 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.382621 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-tls-certs\" (UniqueName: \"kubernetes.io/secret/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-tls-certs\") pod \"fab18d8f-7bf2-44eb-a099-fd2a60990d0a\" (UID: \"fab18d8f-7bf2-44eb-a099-fd2a60990d0a\") " Oct 30 08:04:32.382759 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.382670 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d680a08d-af67-455d-87bc-9fa6a26ad37a-kube-api-access-psh6f\" (UniqueName: \"kubernetes.io/projected/d680a08d-af67-455d-87bc-9fa6a26ad37a-kube-api-access-psh6f\") pod \"d680a08d-af67-455d-87bc-9fa6a26ad37a\" (UID: \"d680a08d-af67-455d-87bc-9fa6a26ad37a\") " Oct 30 08:04:32.382759 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.382701 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d680a08d-af67-455d-87bc-9fa6a26ad37a-home\" (UniqueName: \"kubernetes.io/empty-dir/d680a08d-af67-455d-87bc-9fa6a26ad37a-home\") pod \"d680a08d-af67-455d-87bc-9fa6a26ad37a\" (UID: \"d680a08d-af67-455d-87bc-9fa6a26ad37a\") " Oct 30 08:04:32.383151 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.383122 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d680a08d-af67-455d-87bc-9fa6a26ad37a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d680a08d-af67-455d-87bc-9fa6a26ad37a-dshm\") pod \"d680a08d-af67-455d-87bc-9fa6a26ad37a\" (UID: \"d680a08d-af67-455d-87bc-9fa6a26ad37a\") " Oct 30 08:04:32.383395 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.383372 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-home\" (UniqueName: \"kubernetes.io/empty-dir/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-home\") pod \"61d8d83a-32ea-43b0-9aa7-b70bff2217b8\" (UID: \"61d8d83a-32ea-43b0-9aa7-b70bff2217b8\") " Oct 30 08:04:32.383489 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.383472 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-tls-certs\" (UniqueName: \"kubernetes.io/secret/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-tls-certs\") pod \"61d8d83a-32ea-43b0-9aa7-b70bff2217b8\" (UID: \"61d8d83a-32ea-43b0-9aa7-b70bff2217b8\") " Oct 30 08:04:32.383558 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.383537 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-home\" (UniqueName: \"kubernetes.io/empty-dir/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-home\") pod \"fab18d8f-7bf2-44eb-a099-fd2a60990d0a\" (UID: \"fab18d8f-7bf2-44eb-a099-fd2a60990d0a\") " Oct 30 08:04:32.383655 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.383567 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-tmp-dir\") pod \"61d8d83a-32ea-43b0-9aa7-b70bff2217b8\" (UID: \"61d8d83a-32ea-43b0-9aa7-b70bff2217b8\") " Oct 30 08:04:32.385002 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.384814 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-model-cache" pod "61d8d83a-32ea-43b0-9aa7-b70bff2217b8" (UID: "61d8d83a-32ea-43b0-9aa7-b70bff2217b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:04:32.385138 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.385050 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d680a08d-af67-455d-87bc-9fa6a26ad37a-tmp-dir" pod "d680a08d-af67-455d-87bc-9fa6a26ad37a" (UID: "d680a08d-af67-455d-87bc-9fa6a26ad37a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:04:32.388909 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.386279 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-home" pod "61d8d83a-32ea-43b0-9aa7-b70bff2217b8" (UID: "61d8d83a-32ea-43b0-9aa7-b70bff2217b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:04:32.388909 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.386445 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d680a08d-af67-455d-87bc-9fa6a26ad37a-home" pod "d680a08d-af67-455d-87bc-9fa6a26ad37a" (UID: "d680a08d-af67-455d-87bc-9fa6a26ad37a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:04:32.388909 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.388683 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d680a08d-af67-455d-87bc-9fa6a26ad37a-dshm" pod "d680a08d-af67-455d-87bc-9fa6a26ad37a" (UID: "d680a08d-af67-455d-87bc-9fa6a26ad37a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:04:32.388909 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.388871 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-home" pod "fab18d8f-7bf2-44eb-a099-fd2a60990d0a" (UID: "fab18d8f-7bf2-44eb-a099-fd2a60990d0a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:04:32.389181 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.388975 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-model-cache\") pod \"fab18d8f-7bf2-44eb-a099-fd2a60990d0a\" (UID: \"fab18d8f-7bf2-44eb-a099-fd2a60990d0a\") " Oct 30 08:04:32.389181 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.389066 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d680a08d-af67-455d-87bc-9fa6a26ad37a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d680a08d-af67-455d-87bc-9fa6a26ad37a-model-cache\") pod \"d680a08d-af67-455d-87bc-9fa6a26ad37a\" (UID: \"d680a08d-af67-455d-87bc-9fa6a26ad37a\") " Oct 30 08:04:32.389181 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.389125 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-kube-api-access-qwxx7\" (UniqueName: \"kubernetes.io/projected/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-kube-api-access-qwxx7\") pod \"fab18d8f-7bf2-44eb-a099-fd2a60990d0a\" (UID: \"fab18d8f-7bf2-44eb-a099-fd2a60990d0a\") " Oct 30 08:04:32.389743 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.389329 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-tmp-dir" pod "61d8d83a-32ea-43b0-9aa7-b70bff2217b8" (UID: "61d8d83a-32ea-43b0-9aa7-b70bff2217b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:04:32.389743 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.389466 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-model-cache" pod "fab18d8f-7bf2-44eb-a099-fd2a60990d0a" (UID: "fab18d8f-7bf2-44eb-a099-fd2a60990d0a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:04:32.389743 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.389515 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d680a08d-af67-455d-87bc-9fa6a26ad37a-model-cache" pod "d680a08d-af67-455d-87bc-9fa6a26ad37a" (UID: "d680a08d-af67-455d-87bc-9fa6a26ad37a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:04:32.389743 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.389595 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-tls-certs" pod "fab18d8f-7bf2-44eb-a099-fd2a60990d0a" (UID: "fab18d8f-7bf2-44eb-a099-fd2a60990d0a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:04:32.389959 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.389937 4951 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d680a08d-af67-455d-87bc-9fa6a26ad37a-dshm\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:32.390017 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.389966 4951 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-home\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:32.390017 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.389980 4951 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-home\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:32.390017 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.389993 4951 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-tmp-dir\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:32.390017 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.390007 4951 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-model-cache\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:32.390226 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.390022 4951 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d680a08d-af67-455d-87bc-9fa6a26ad37a-model-cache\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:32.390226 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.390036 4951 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-tmp-dir\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:32.390226 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.390048 4951 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-model-cache\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:32.390226 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.390061 4951 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d680a08d-af67-455d-87bc-9fa6a26ad37a-tmp-dir\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:32.390226 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.390073 4951 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-tls-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:32.390226 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.390085 4951 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d680a08d-af67-455d-87bc-9fa6a26ad37a-home\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:32.390687 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.390661 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-kube-api-access-n89sr" pod "61d8d83a-32ea-43b0-9aa7-b70bff2217b8" (UID: "61d8d83a-32ea-43b0-9aa7-b70bff2217b8"). InnerVolumeSpecName "kube-api-access-n89sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:04:32.390983 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.390918 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d680a08d-af67-455d-87bc-9fa6a26ad37a-tls-certs" pod "d680a08d-af67-455d-87bc-9fa6a26ad37a" (UID: "d680a08d-af67-455d-87bc-9fa6a26ad37a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:04:32.390983 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.390929 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-dshm" pod "61d8d83a-32ea-43b0-9aa7-b70bff2217b8" (UID: "61d8d83a-32ea-43b0-9aa7-b70bff2217b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:04:32.391706 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.391680 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-tls-certs" pod "61d8d83a-32ea-43b0-9aa7-b70bff2217b8" (UID: "61d8d83a-32ea-43b0-9aa7-b70bff2217b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:04:32.391836 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.391812 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-dshm" pod "fab18d8f-7bf2-44eb-a099-fd2a60990d0a" (UID: "fab18d8f-7bf2-44eb-a099-fd2a60990d0a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:04:32.392941 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.392543 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-kube-api-access-qwxx7" pod "fab18d8f-7bf2-44eb-a099-fd2a60990d0a" (UID: "fab18d8f-7bf2-44eb-a099-fd2a60990d0a"). InnerVolumeSpecName "kube-api-access-qwxx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:04:32.393227 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.393129 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d680a08d-af67-455d-87bc-9fa6a26ad37a-kube-api-access-psh6f" pod "d680a08d-af67-455d-87bc-9fa6a26ad37a" (UID: "d680a08d-af67-455d-87bc-9fa6a26ad37a"). InnerVolumeSpecName "kube-api-access-psh6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:04:32.492252 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.492158 4951 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-tls-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:32.492252 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.492203 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwxx7\" (UniqueName: \"kubernetes.io/projected/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-kube-api-access-qwxx7\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:32.492252 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.492219 4951 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d680a08d-af67-455d-87bc-9fa6a26ad37a-tls-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:32.492252 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.492232 4951 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-dshm\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:32.492252 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.492246 4951 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fab18d8f-7bf2-44eb-a099-fd2a60990d0a-dshm\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:32.492554 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.492262 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n89sr\" (UniqueName: \"kubernetes.io/projected/61d8d83a-32ea-43b0-9aa7-b70bff2217b8-kube-api-access-n89sr\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:32.492554 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:32.492275 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psh6f\" (UniqueName: \"kubernetes.io/projected/d680a08d-af67-455d-87bc-9fa6a26ad37a-kube-api-access-psh6f\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:04:33.236666 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:33.217344 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 08:04:33.236666 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:33.231731 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" Oct 30 08:04:33.240622 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:33.237903 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk" event={"ID":"d680a08d-af67-455d-87bc-9fa6a26ad37a","Type":"ContainerDied","Data":"cc98cb2d8be4411ca0ac05d5341722730f3b572e9f8bc03cf3bbfb69a5b37154"} Oct 30 08:04:33.240622 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:33.237965 4951 scope.go:122] "RemoveContainer" containerID="6c4fa67a100d7e4764ab97b771c3999de42eac25335905cf81ca9a426c19b8fe" Oct 30 08:04:33.255653 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:33.255174 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" event={"ID":"61d8d83a-32ea-43b0-9aa7-b70bff2217b8","Type":"ContainerDied","Data":"af0fbaaaf46a9cd8fcc0c0f75b947f1e33745a3216331442604f03b589820977"} Oct 30 08:04:33.255653 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:33.255233 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv" Oct 30 08:04:33.261421 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:33.261387 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" event={"ID":"fab18d8f-7bf2-44eb-a099-fd2a60990d0a","Type":"ContainerDied","Data":"5dcd17ffeec2036f8fd7616817f705b12c04e1ff93ad6e0ae0dce52280ee6eaa"} Oct 30 08:04:33.261618 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:33.261545 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct" Oct 30 08:04:33.269733 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:33.269705 4951 scope.go:122] "RemoveContainer" containerID="449c36e67c4891a77e26ef1ed03c3ca0eebfd10ef835103293fc97656d434e5e" Oct 30 08:04:33.288592 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:33.288560 4951 scope.go:122] "RemoveContainer" containerID="7f032fb28d453ab0b355d92f034f3bb800e34c6034836b17077a71166a4d56ef" Oct 30 08:04:33.316715 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:33.316688 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk"] Oct 30 08:04:33.328901 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:33.328879 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b72d79/premium-w0-b72d79-kserve-7d9886d56f-dfwsk"] Oct 30 08:04:33.374763 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:33.374731 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct"] Oct 30 08:04:33.381731 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:33.381699 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b72d79/simulator-w0-b72d79-kserve-697678f9b-vmbct"] Oct 30 08:04:33.424392 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:33.424358 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv"] Oct 30 08:04:33.427006 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:33.426981 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b72d79/distinct-2-w0-b72d79-kserve-668bf7d4c9-fghbv"] Oct 30 08:04:33.773400 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:33.773368 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" path="/var/lib/kubelet/pods/61d8d83a-32ea-43b0-9aa7-b70bff2217b8/volumes" Oct 30 08:04:33.773917 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:33.773901 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" path="/var/lib/kubelet/pods/d680a08d-af67-455d-87bc-9fa6a26ad37a/volumes" Oct 30 08:04:33.774299 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:04:33.774286 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" path="/var/lib/kubelet/pods/fab18d8f-7bf2-44eb-a099-fd2a60990d0a/volumes" Oct 30 08:04:53.024633 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:04:53.024589 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 08:05:09.526237 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:09.526197 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4"] Oct 30 08:05:09.526791 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:09.526466 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4" podUID="bb29167c-8d9b-4938-afb5-0c974e7e34e7" containerName="maas-api" containerID="cri-o://da0add70e687fc05490cbe8469cba5140027d4edde3f71c0b6448b23f1ca5fe5" gracePeriod=30 Oct 30 08:05:09.880431 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:09.880396 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4" Oct 30 08:05:09.972593 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:09.961940 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb29167c-8d9b-4938-afb5-0c974e7e34e7-kube-api-access-w9mct\" (UniqueName: \"kubernetes.io/projected/bb29167c-8d9b-4938-afb5-0c974e7e34e7-kube-api-access-w9mct\") pod \"bb29167c-8d9b-4938-afb5-0c974e7e34e7\" (UID: \"bb29167c-8d9b-4938-afb5-0c974e7e34e7\") " Oct 30 08:05:09.972593 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:09.962061 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb29167c-8d9b-4938-afb5-0c974e7e34e7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb29167c-8d9b-4938-afb5-0c974e7e34e7-openshift-service-ca\") pod \"bb29167c-8d9b-4938-afb5-0c974e7e34e7\" (UID: \"bb29167c-8d9b-4938-afb5-0c974e7e34e7\") " Oct 30 08:05:09.972593 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:09.962179 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb29167c-8d9b-4938-afb5-0c974e7e34e7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb29167c-8d9b-4938-afb5-0c974e7e34e7-metrics-certs\") pod \"bb29167c-8d9b-4938-afb5-0c974e7e34e7\" (UID: \"bb29167c-8d9b-4938-afb5-0c974e7e34e7\") " Oct 30 08:05:09.972593 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:09.962252 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb29167c-8d9b-4938-afb5-0c974e7e34e7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb29167c-8d9b-4938-afb5-0c974e7e34e7-maas-api-tls\") pod \"bb29167c-8d9b-4938-afb5-0c974e7e34e7\" (UID: \"bb29167c-8d9b-4938-afb5-0c974e7e34e7\") " Oct 30 08:05:09.972593 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:09.971056 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb29167c-8d9b-4938-afb5-0c974e7e34e7-openshift-service-ca" pod "bb29167c-8d9b-4938-afb5-0c974e7e34e7" (UID: "bb29167c-8d9b-4938-afb5-0c974e7e34e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:05:10.001375 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:09.988284 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb29167c-8d9b-4938-afb5-0c974e7e34e7-kube-api-access-w9mct" pod "bb29167c-8d9b-4938-afb5-0c974e7e34e7" (UID: "bb29167c-8d9b-4938-afb5-0c974e7e34e7"). InnerVolumeSpecName "kube-api-access-w9mct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:05:10.001375 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:09.988432 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb29167c-8d9b-4938-afb5-0c974e7e34e7-metrics-certs" pod "bb29167c-8d9b-4938-afb5-0c974e7e34e7" (UID: "bb29167c-8d9b-4938-afb5-0c974e7e34e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:05:10.001375 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:09.995500 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb29167c-8d9b-4938-afb5-0c974e7e34e7-maas-api-tls" pod "bb29167c-8d9b-4938-afb5-0c974e7e34e7" (UID: "bb29167c-8d9b-4938-afb5-0c974e7e34e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:05:10.072605 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:10.070235 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb29167c-8d9b-4938-afb5-0c974e7e34e7-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:05:10.072605 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:10.070278 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb29167c-8d9b-4938-afb5-0c974e7e34e7-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:05:10.072605 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:10.070294 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9mct\" (UniqueName: \"kubernetes.io/projected/bb29167c-8d9b-4938-afb5-0c974e7e34e7-kube-api-access-w9mct\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:05:10.072605 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:10.070309 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb29167c-8d9b-4938-afb5-0c974e7e34e7-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:05:10.560977 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:10.560938 4951 generic.go:356] "Generic (PLEG): container finished" podID="bb29167c-8d9b-4938-afb5-0c974e7e34e7" containerID="da0add70e687fc05490cbe8469cba5140027d4edde3f71c0b6448b23f1ca5fe5" exitCode=0 Oct 30 08:05:10.561441 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:10.561015 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4" Oct 30 08:05:10.561441 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:10.561021 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4" event={"ID":"bb29167c-8d9b-4938-afb5-0c974e7e34e7","Type":"ContainerDied","Data":"da0add70e687fc05490cbe8469cba5140027d4edde3f71c0b6448b23f1ca5fe5"} Oct 30 08:05:10.561441 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:10.561119 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4" event={"ID":"bb29167c-8d9b-4938-afb5-0c974e7e34e7","Type":"ContainerDied","Data":"bf7b95cdf69ca8af9e3cd59d02cf6f3c3e77188b06b4bab8022acad936e07950"} Oct 30 08:05:10.561441 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:10.561143 4951 scope.go:122] "RemoveContainer" containerID="da0add70e687fc05490cbe8469cba5140027d4edde3f71c0b6448b23f1ca5fe5" Oct 30 08:05:10.572618 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:10.572593 4951 scope.go:122] "RemoveContainer" containerID="da0add70e687fc05490cbe8469cba5140027d4edde3f71c0b6448b23f1ca5fe5" Oct 30 08:05:10.572955 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:05:10.572931 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da0add70e687fc05490cbe8469cba5140027d4edde3f71c0b6448b23f1ca5fe5\": container with ID starting with da0add70e687fc05490cbe8469cba5140027d4edde3f71c0b6448b23f1ca5fe5 not found: ID does not exist" containerID="da0add70e687fc05490cbe8469cba5140027d4edde3f71c0b6448b23f1ca5fe5" Oct 30 08:05:10.573075 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:10.572966 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da0add70e687fc05490cbe8469cba5140027d4edde3f71c0b6448b23f1ca5fe5"} err="failed to get container status \"da0add70e687fc05490cbe8469cba5140027d4edde3f71c0b6448b23f1ca5fe5\": rpc error: code = NotFound desc = could not find container \"da0add70e687fc05490cbe8469cba5140027d4edde3f71c0b6448b23f1ca5fe5\": container with ID starting with da0add70e687fc05490cbe8469cba5140027d4edde3f71c0b6448b23f1ca5fe5 not found: ID does not exist" Oct 30 08:05:10.616893 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:10.616861 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4"] Oct 30 08:05:10.621314 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:10.621290 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b72d79-7f9c7b97b-cfft4"] Oct 30 08:05:11.772224 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:11.772190 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb29167c-8d9b-4938-afb5-0c974e7e34e7" path="/var/lib/kubelet/pods/bb29167c-8d9b-4938-afb5-0c974e7e34e7/volumes" Oct 30 08:05:15.246323 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:15.244074 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx"] Oct 30 08:05:15.246323 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:15.244300 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" podUID="e684243f-7d08-4ae2-b946-7cea93656322" containerName="istio-proxy" containerID="cri-o://afcc967650aa7c9458537c5e2ccf39b40198c43f1836c91385a87512148ee5af" gracePeriod=30 Oct 30 08:05:17.617203 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:17.617160 4951 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 08:05:17.617630 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:17.617240 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" podUID="e684243f-7d08-4ae2-b946-7cea93656322" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 08:05:20.591918 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.591888 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 08:05:20.630460 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.630424 4951 generic.go:356] "Generic (PLEG): container finished" podID="e684243f-7d08-4ae2-b946-7cea93656322" containerID="afcc967650aa7c9458537c5e2ccf39b40198c43f1836c91385a87512148ee5af" exitCode=0 Oct 30 08:05:20.630651 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.630485 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" Oct 30 08:05:20.630651 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.630503 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" event={"ID":"e684243f-7d08-4ae2-b946-7cea93656322","Type":"ContainerDied","Data":"afcc967650aa7c9458537c5e2ccf39b40198c43f1836c91385a87512148ee5af"} Oct 30 08:05:20.630651 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.630566 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx" event={"ID":"e684243f-7d08-4ae2-b946-7cea93656322","Type":"ContainerDied","Data":"3be644bfca3483029a89370d935548c07834a02d214fc2e8ca8870e76e2f70df"} Oct 30 08:05:20.630651 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.630618 4951 scope.go:122] "RemoveContainer" containerID="afcc967650aa7c9458537c5e2ccf39b40198c43f1836c91385a87512148ee5af" Oct 30 08:05:20.642643 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.642615 4951 scope.go:122] "RemoveContainer" containerID="dbc3d6ada8ab2566f28b0e170bc9eb549a31631e898acb0ea91e6de31d6e7aef" Oct 30 08:05:20.655012 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.654988 4951 scope.go:122] "RemoveContainer" containerID="afcc967650aa7c9458537c5e2ccf39b40198c43f1836c91385a87512148ee5af" Oct 30 08:05:20.655385 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:05:20.655361 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afcc967650aa7c9458537c5e2ccf39b40198c43f1836c91385a87512148ee5af\": container with ID starting with afcc967650aa7c9458537c5e2ccf39b40198c43f1836c91385a87512148ee5af not found: ID does not exist" containerID="afcc967650aa7c9458537c5e2ccf39b40198c43f1836c91385a87512148ee5af" Oct 30 08:05:20.655493 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.655398 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afcc967650aa7c9458537c5e2ccf39b40198c43f1836c91385a87512148ee5af"} err="failed to get container status \"afcc967650aa7c9458537c5e2ccf39b40198c43f1836c91385a87512148ee5af\": rpc error: code = NotFound desc = could not find container \"afcc967650aa7c9458537c5e2ccf39b40198c43f1836c91385a87512148ee5af\": container with ID starting with afcc967650aa7c9458537c5e2ccf39b40198c43f1836c91385a87512148ee5af not found: ID does not exist" Oct 30 08:05:20.655493 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.655422 4951 scope.go:122] "RemoveContainer" containerID="dbc3d6ada8ab2566f28b0e170bc9eb549a31631e898acb0ea91e6de31d6e7aef" Oct 30 08:05:20.656001 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:05:20.655960 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbc3d6ada8ab2566f28b0e170bc9eb549a31631e898acb0ea91e6de31d6e7aef\": container with ID starting with dbc3d6ada8ab2566f28b0e170bc9eb549a31631e898acb0ea91e6de31d6e7aef not found: ID does not exist" containerID="dbc3d6ada8ab2566f28b0e170bc9eb549a31631e898acb0ea91e6de31d6e7aef" Oct 30 08:05:20.656102 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.655994 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbc3d6ada8ab2566f28b0e170bc9eb549a31631e898acb0ea91e6de31d6e7aef"} err="failed to get container status \"dbc3d6ada8ab2566f28b0e170bc9eb549a31631e898acb0ea91e6de31d6e7aef\": rpc error: code = NotFound desc = could not find container \"dbc3d6ada8ab2566f28b0e170bc9eb549a31631e898acb0ea91e6de31d6e7aef\": container with ID starting with dbc3d6ada8ab2566f28b0e170bc9eb549a31631e898acb0ea91e6de31d6e7aef not found: ID does not exist" Oct 30 08:05:20.715490 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.715457 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-workload-certs\") pod \"e684243f-7d08-4ae2-b946-7cea93656322\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " Oct 30 08:05:20.715664 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.715500 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e684243f-7d08-4ae2-b946-7cea93656322-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e684243f-7d08-4ae2-b946-7cea93656322-istiod-ca-cert\") pod \"e684243f-7d08-4ae2-b946-7cea93656322\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " Oct 30 08:05:20.715664 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.715598 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e684243f-7d08-4ae2-b946-7cea93656322-istio-token\" (UniqueName: \"kubernetes.io/projected/e684243f-7d08-4ae2-b946-7cea93656322-istio-token\") pod \"e684243f-7d08-4ae2-b946-7cea93656322\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " Oct 30 08:05:20.715664 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.715640 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e684243f-7d08-4ae2-b946-7cea93656322-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e684243f-7d08-4ae2-b946-7cea93656322-istio-podinfo\") pod \"e684243f-7d08-4ae2-b946-7cea93656322\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " Oct 30 08:05:20.715837 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.715733 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-credential-socket\") pod \"e684243f-7d08-4ae2-b946-7cea93656322\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " Oct 30 08:05:20.715837 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.715775 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-workload-certs" pod "e684243f-7d08-4ae2-b946-7cea93656322" (UID: "e684243f-7d08-4ae2-b946-7cea93656322"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:05:20.715837 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.715798 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-istio-envoy\") pod \"e684243f-7d08-4ae2-b946-7cea93656322\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " Oct 30 08:05:20.716008 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.715855 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-workload-socket\") pod \"e684243f-7d08-4ae2-b946-7cea93656322\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " Oct 30 08:05:20.716008 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.715882 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e684243f-7d08-4ae2-b946-7cea93656322-kube-api-access-bbdd8\" (UniqueName: \"kubernetes.io/projected/e684243f-7d08-4ae2-b946-7cea93656322-kube-api-access-bbdd8\") pod \"e684243f-7d08-4ae2-b946-7cea93656322\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " Oct 30 08:05:20.716008 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.715933 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-istio-data\") pod \"e684243f-7d08-4ae2-b946-7cea93656322\" (UID: \"e684243f-7d08-4ae2-b946-7cea93656322\") " Oct 30 08:05:20.716175 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.715947 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e684243f-7d08-4ae2-b946-7cea93656322-istiod-ca-cert" pod "e684243f-7d08-4ae2-b946-7cea93656322" (UID: "e684243f-7d08-4ae2-b946-7cea93656322"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:05:20.716175 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.716042 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-credential-socket" pod "e684243f-7d08-4ae2-b946-7cea93656322" (UID: "e684243f-7d08-4ae2-b946-7cea93656322"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:05:20.716175 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.716105 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-workload-socket" pod "e684243f-7d08-4ae2-b946-7cea93656322" (UID: "e684243f-7d08-4ae2-b946-7cea93656322"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:05:20.716303 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.716289 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-istio-data" pod "e684243f-7d08-4ae2-b946-7cea93656322" (UID: "e684243f-7d08-4ae2-b946-7cea93656322"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:05:20.716493 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.716477 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:05:20.716629 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.716495 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:05:20.716629 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.716509 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:05:20.716629 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.716522 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e684243f-7d08-4ae2-b946-7cea93656322-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:05:20.716629 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.716535 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:05:20.718256 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.718228 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e684243f-7d08-4ae2-b946-7cea93656322-istio-token" pod "e684243f-7d08-4ae2-b946-7cea93656322" (UID: "e684243f-7d08-4ae2-b946-7cea93656322"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:05:20.718256 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.718248 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e684243f-7d08-4ae2-b946-7cea93656322-kube-api-access-bbdd8" pod "e684243f-7d08-4ae2-b946-7cea93656322" (UID: "e684243f-7d08-4ae2-b946-7cea93656322"). InnerVolumeSpecName "kube-api-access-bbdd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:05:20.718517 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.718490 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-istio-envoy" pod "e684243f-7d08-4ae2-b946-7cea93656322" (UID: "e684243f-7d08-4ae2-b946-7cea93656322"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:05:20.718667 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.718494 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e684243f-7d08-4ae2-b946-7cea93656322-istio-podinfo" pod "e684243f-7d08-4ae2-b946-7cea93656322" (UID: "e684243f-7d08-4ae2-b946-7cea93656322"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:05:20.817969 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.817872 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e684243f-7d08-4ae2-b946-7cea93656322-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:05:20.817969 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.817912 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbdd8\" (UniqueName: \"kubernetes.io/projected/e684243f-7d08-4ae2-b946-7cea93656322-kube-api-access-bbdd8\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:05:20.817969 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.817921 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e684243f-7d08-4ae2-b946-7cea93656322-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:05:20.817969 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.817930 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e684243f-7d08-4ae2-b946-7cea93656322-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:05:20.982021 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.981982 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx"] Oct 30 08:05:20.994145 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:20.994108 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-8qmcx"] Oct 30 08:05:21.774161 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:05:21.774120 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e684243f-7d08-4ae2-b946-7cea93656322" path="/var/lib/kubelet/pods/e684243f-7d08-4ae2-b946-7cea93656322/volumes" Oct 30 08:05:23.023328 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:05:23.023292 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 08:05:53.025560 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:05:53.025516 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 08:06:06.275960 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:06.275922 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b"] Oct 30 08:06:06.276523 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:06.276165 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" podUID="705d9fa4-a31c-4fc9-b27b-48874576f32a" containerName="istio-proxy" containerID="cri-o://7e20545d01605913c47db9d44daac8ea73e912bd464bd6c46e9fc78bd0293a2b" gracePeriod=30 Oct 30 08:06:11.567051 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.566970 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 08:06:11.667436 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.667398 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-workload-certs\") pod \"705d9fa4-a31c-4fc9-b27b-48874576f32a\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " Oct 30 08:06:11.667643 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.667508 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/705d9fa4-a31c-4fc9-b27b-48874576f32a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/705d9fa4-a31c-4fc9-b27b-48874576f32a-istiod-ca-cert\") pod \"705d9fa4-a31c-4fc9-b27b-48874576f32a\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " Oct 30 08:06:11.667643 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.667536 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/705d9fa4-a31c-4fc9-b27b-48874576f32a-kube-api-access-f4hhx\" (UniqueName: \"kubernetes.io/projected/705d9fa4-a31c-4fc9-b27b-48874576f32a-kube-api-access-f4hhx\") pod \"705d9fa4-a31c-4fc9-b27b-48874576f32a\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " Oct 30 08:06:11.667643 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.667614 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/705d9fa4-a31c-4fc9-b27b-48874576f32a-istio-token\" (UniqueName: \"kubernetes.io/projected/705d9fa4-a31c-4fc9-b27b-48874576f32a-istio-token\") pod \"705d9fa4-a31c-4fc9-b27b-48874576f32a\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " Oct 30 08:06:11.667804 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.667653 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-istio-envoy\") pod \"705d9fa4-a31c-4fc9-b27b-48874576f32a\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " Oct 30 08:06:11.667804 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.667746 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-workload-socket\") pod \"705d9fa4-a31c-4fc9-b27b-48874576f32a\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " Oct 30 08:06:11.667804 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.667781 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/705d9fa4-a31c-4fc9-b27b-48874576f32a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/705d9fa4-a31c-4fc9-b27b-48874576f32a-istio-podinfo\") pod \"705d9fa4-a31c-4fc9-b27b-48874576f32a\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " Oct 30 08:06:11.667994 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.667842 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-istio-data\") pod \"705d9fa4-a31c-4fc9-b27b-48874576f32a\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " Oct 30 08:06:11.667994 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.667938 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-credential-socket\") pod \"705d9fa4-a31c-4fc9-b27b-48874576f32a\" (UID: \"705d9fa4-a31c-4fc9-b27b-48874576f32a\") " Oct 30 08:06:11.668770 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.668744 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-credential-socket" pod "705d9fa4-a31c-4fc9-b27b-48874576f32a" (UID: "705d9fa4-a31c-4fc9-b27b-48874576f32a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:11.668968 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.668947 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-workload-certs" pod "705d9fa4-a31c-4fc9-b27b-48874576f32a" (UID: "705d9fa4-a31c-4fc9-b27b-48874576f32a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:11.669920 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.669823 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/705d9fa4-a31c-4fc9-b27b-48874576f32a-istiod-ca-cert" pod "705d9fa4-a31c-4fc9-b27b-48874576f32a" (UID: "705d9fa4-a31c-4fc9-b27b-48874576f32a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:06:11.670388 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.670340 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-workload-socket" pod "705d9fa4-a31c-4fc9-b27b-48874576f32a" (UID: "705d9fa4-a31c-4fc9-b27b-48874576f32a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:11.670945 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.670209 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-istio-data" pod "705d9fa4-a31c-4fc9-b27b-48874576f32a" (UID: "705d9fa4-a31c-4fc9-b27b-48874576f32a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:11.672260 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.672224 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-istio-envoy" pod "705d9fa4-a31c-4fc9-b27b-48874576f32a" (UID: "705d9fa4-a31c-4fc9-b27b-48874576f32a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:11.673460 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.673434 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/705d9fa4-a31c-4fc9-b27b-48874576f32a-kube-api-access-f4hhx" pod "705d9fa4-a31c-4fc9-b27b-48874576f32a" (UID: "705d9fa4-a31c-4fc9-b27b-48874576f32a"). InnerVolumeSpecName "kube-api-access-f4hhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:06:11.673748 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.673709 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/705d9fa4-a31c-4fc9-b27b-48874576f32a-istio-token" pod "705d9fa4-a31c-4fc9-b27b-48874576f32a" (UID: "705d9fa4-a31c-4fc9-b27b-48874576f32a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:06:11.674009 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.673978 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/705d9fa4-a31c-4fc9-b27b-48874576f32a-istio-podinfo" pod "705d9fa4-a31c-4fc9-b27b-48874576f32a" (UID: "705d9fa4-a31c-4fc9-b27b-48874576f32a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:06:11.769476 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.769426 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/705d9fa4-a31c-4fc9-b27b-48874576f32a-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.769476 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.769460 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4hhx\" (UniqueName: \"kubernetes.io/projected/705d9fa4-a31c-4fc9-b27b-48874576f32a-kube-api-access-f4hhx\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.769476 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.769475 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/705d9fa4-a31c-4fc9-b27b-48874576f32a-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.769815 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.769486 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.769815 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.769498 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.769815 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.769509 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/705d9fa4-a31c-4fc9-b27b-48874576f32a-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.769815 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.769520 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.769815 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.769531 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.769815 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.769544 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/705d9fa4-a31c-4fc9-b27b-48874576f32a-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.974567 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.974476 4951 generic.go:356] "Generic (PLEG): container finished" podID="705d9fa4-a31c-4fc9-b27b-48874576f32a" containerID="7e20545d01605913c47db9d44daac8ea73e912bd464bd6c46e9fc78bd0293a2b" exitCode=0 Oct 30 08:06:11.974567 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.974531 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" Oct 30 08:06:11.974567 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.974560 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" event={"ID":"705d9fa4-a31c-4fc9-b27b-48874576f32a","Type":"ContainerDied","Data":"7e20545d01605913c47db9d44daac8ea73e912bd464bd6c46e9fc78bd0293a2b"} Oct 30 08:06:11.974879 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.974625 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b" event={"ID":"705d9fa4-a31c-4fc9-b27b-48874576f32a","Type":"ContainerDied","Data":"5da7996fb4e739a51d2b725b24d4d0b826fdf22ed7eed4fa2f9ccf5ba0b804c5"} Oct 30 08:06:11.974879 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.974650 4951 scope.go:122] "RemoveContainer" containerID="7e20545d01605913c47db9d44daac8ea73e912bd464bd6c46e9fc78bd0293a2b" Oct 30 08:06:11.984639 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.984622 4951 scope.go:122] "RemoveContainer" containerID="7e20545d01605913c47db9d44daac8ea73e912bd464bd6c46e9fc78bd0293a2b" Oct 30 08:06:11.984942 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:06:11.984924 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e20545d01605913c47db9d44daac8ea73e912bd464bd6c46e9fc78bd0293a2b\": container with ID starting with 7e20545d01605913c47db9d44daac8ea73e912bd464bd6c46e9fc78bd0293a2b not found: ID does not exist" containerID="7e20545d01605913c47db9d44daac8ea73e912bd464bd6c46e9fc78bd0293a2b" Oct 30 08:06:11.984996 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:11.984950 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e20545d01605913c47db9d44daac8ea73e912bd464bd6c46e9fc78bd0293a2b"} err="failed to get container status \"7e20545d01605913c47db9d44daac8ea73e912bd464bd6c46e9fc78bd0293a2b\": rpc error: code = NotFound desc = could not find container \"7e20545d01605913c47db9d44daac8ea73e912bd464bd6c46e9fc78bd0293a2b\": container with ID starting with 7e20545d01605913c47db9d44daac8ea73e912bd464bd6c46e9fc78bd0293a2b not found: ID does not exist" Oct 30 08:06:12.028690 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:12.028665 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b"] Oct 30 08:06:12.040739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:12.040706 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xx84b"] Oct 30 08:06:13.774172 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:13.774136 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="705d9fa4-a31c-4fc9-b27b-48874576f32a" path="/var/lib/kubelet/pods/705d9fa4-a31c-4fc9-b27b-48874576f32a/volumes" Oct 30 08:06:16.005886 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:16.005805 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz"] Oct 30 08:06:16.006298 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:16.006000 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz" podUID="d729858d-5129-43e9-b6fe-129b8267791e" containerName="discovery" containerID="cri-o://c8a3f9c150e1e5615497e53697ab440f07852e30a824f9676714da6fecf2b28e" gracePeriod=30 Oct 30 08:06:16.296801 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:16.296771 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz" Oct 30 08:06:16.331515 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:16.331465 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d729858d-5129-43e9-b6fe-129b8267791e-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d729858d-5129-43e9-b6fe-129b8267791e-istio-csr-dns-cert\") pod \"d729858d-5129-43e9-b6fe-129b8267791e\" (UID: \"d729858d-5129-43e9-b6fe-129b8267791e\") " Oct 30 08:06:16.331776 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:16.331625 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d729858d-5129-43e9-b6fe-129b8267791e-cacerts\" (UniqueName: \"kubernetes.io/secret/d729858d-5129-43e9-b6fe-129b8267791e-cacerts\") pod \"d729858d-5129-43e9-b6fe-129b8267791e\" (UID: \"d729858d-5129-43e9-b6fe-129b8267791e\") " Oct 30 08:06:16.331776 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:16.331668 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d729858d-5129-43e9-b6fe-129b8267791e-kube-api-access-99x2r\" (UniqueName: \"kubernetes.io/projected/d729858d-5129-43e9-b6fe-129b8267791e-kube-api-access-99x2r\") pod \"d729858d-5129-43e9-b6fe-129b8267791e\" (UID: \"d729858d-5129-43e9-b6fe-129b8267791e\") " Oct 30 08:06:16.331776 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:16.331756 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d729858d-5129-43e9-b6fe-129b8267791e-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d729858d-5129-43e9-b6fe-129b8267791e-istio-csr-ca-configmap\") pod \"d729858d-5129-43e9-b6fe-129b8267791e\" (UID: \"d729858d-5129-43e9-b6fe-129b8267791e\") " Oct 30 08:06:16.331984 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:16.331816 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d729858d-5129-43e9-b6fe-129b8267791e-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d729858d-5129-43e9-b6fe-129b8267791e-local-certs\") pod \"d729858d-5129-43e9-b6fe-129b8267791e\" (UID: \"d729858d-5129-43e9-b6fe-129b8267791e\") " Oct 30 08:06:16.331984 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:16.331857 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d729858d-5129-43e9-b6fe-129b8267791e-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d729858d-5129-43e9-b6fe-129b8267791e-istio-kubeconfig\") pod \"d729858d-5129-43e9-b6fe-129b8267791e\" (UID: \"d729858d-5129-43e9-b6fe-129b8267791e\") " Oct 30 08:06:16.331984 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:16.331881 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d729858d-5129-43e9-b6fe-129b8267791e-istio-token\" (UniqueName: \"kubernetes.io/projected/d729858d-5129-43e9-b6fe-129b8267791e-istio-token\") pod \"d729858d-5129-43e9-b6fe-129b8267791e\" (UID: \"d729858d-5129-43e9-b6fe-129b8267791e\") " Oct 30 08:06:16.333007 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:16.332872 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d729858d-5129-43e9-b6fe-129b8267791e-istio-csr-ca-configmap" pod "d729858d-5129-43e9-b6fe-129b8267791e" (UID: "d729858d-5129-43e9-b6fe-129b8267791e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:06:16.335149 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:16.334168 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d729858d-5129-43e9-b6fe-129b8267791e-istio-csr-dns-cert" pod "d729858d-5129-43e9-b6fe-129b8267791e" (UID: "d729858d-5129-43e9-b6fe-129b8267791e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:06:16.335149 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:16.334367 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d729858d-5129-43e9-b6fe-129b8267791e-cacerts" pod "d729858d-5129-43e9-b6fe-129b8267791e" (UID: "d729858d-5129-43e9-b6fe-129b8267791e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:06:16.335149 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:16.334499 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d729858d-5129-43e9-b6fe-129b8267791e-kube-api-access-99x2r" pod "d729858d-5129-43e9-b6fe-129b8267791e" (UID: "d729858d-5129-43e9-b6fe-129b8267791e"). InnerVolumeSpecName "kube-api-access-99x2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:06:16.335149 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:16.334720 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d729858d-5129-43e9-b6fe-129b8267791e-istio-kubeconfig" pod "d729858d-5129-43e9-b6fe-129b8267791e" (UID: "d729858d-5129-43e9-b6fe-129b8267791e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:06:16.335149 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:16.335120 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d729858d-5129-43e9-b6fe-129b8267791e-local-certs" pod "d729858d-5129-43e9-b6fe-129b8267791e" (UID: "d729858d-5129-43e9-b6fe-129b8267791e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:16.335417 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:16.335362 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d729858d-5129-43e9-b6fe-129b8267791e-istio-token" pod "d729858d-5129-43e9-b6fe-129b8267791e" (UID: "d729858d-5129-43e9-b6fe-129b8267791e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:06:16.433439 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:16.433403 4951 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d729858d-5129-43e9-b6fe-129b8267791e-cacerts\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:06:16.433439 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:16.433433 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99x2r\" (UniqueName: \"kubernetes.io/projected/d729858d-5129-43e9-b6fe-129b8267791e-kube-api-access-99x2r\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:06:16.433439 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:16.433445 4951 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d729858d-5129-43e9-b6fe-129b8267791e-istio-csr-ca-configmap\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:06:16.433724 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:16.433454 4951 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d729858d-5129-43e9-b6fe-129b8267791e-local-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:06:16.433724 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:16.433462 4951 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d729858d-5129-43e9-b6fe-129b8267791e-istio-kubeconfig\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:06:16.433724 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:16.433470 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d729858d-5129-43e9-b6fe-129b8267791e-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:06:16.433724 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:16.433478 4951 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d729858d-5129-43e9-b6fe-129b8267791e-istio-csr-dns-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:06:17.008279 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:17.008244 4951 generic.go:356] "Generic (PLEG): container finished" podID="d729858d-5129-43e9-b6fe-129b8267791e" containerID="c8a3f9c150e1e5615497e53697ab440f07852e30a824f9676714da6fecf2b28e" exitCode=0 Oct 30 08:06:17.008747 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:17.008285 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz" Oct 30 08:06:17.008747 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:17.008297 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz" event={"ID":"d729858d-5129-43e9-b6fe-129b8267791e","Type":"ContainerDied","Data":"c8a3f9c150e1e5615497e53697ab440f07852e30a824f9676714da6fecf2b28e"} Oct 30 08:06:17.008747 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:17.008336 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz" event={"ID":"d729858d-5129-43e9-b6fe-129b8267791e","Type":"ContainerDied","Data":"39d12021cc7e1038f824a2f2fa7a53561ab6e6e0ae8a554f24fe70773bda2bb0"} Oct 30 08:06:17.008747 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:17.008354 4951 scope.go:122] "RemoveContainer" containerID="c8a3f9c150e1e5615497e53697ab440f07852e30a824f9676714da6fecf2b28e" Oct 30 08:06:17.024938 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:17.024916 4951 scope.go:122] "RemoveContainer" containerID="c8a3f9c150e1e5615497e53697ab440f07852e30a824f9676714da6fecf2b28e" Oct 30 08:06:17.025254 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:06:17.025233 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8a3f9c150e1e5615497e53697ab440f07852e30a824f9676714da6fecf2b28e\": container with ID starting with c8a3f9c150e1e5615497e53697ab440f07852e30a824f9676714da6fecf2b28e not found: ID does not exist" containerID="c8a3f9c150e1e5615497e53697ab440f07852e30a824f9676714da6fecf2b28e" Oct 30 08:06:17.025328 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:17.025263 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8a3f9c150e1e5615497e53697ab440f07852e30a824f9676714da6fecf2b28e"} err="failed to get container status \"c8a3f9c150e1e5615497e53697ab440f07852e30a824f9676714da6fecf2b28e\": rpc error: code = NotFound desc = could not find container \"c8a3f9c150e1e5615497e53697ab440f07852e30a824f9676714da6fecf2b28e\": container with ID starting with c8a3f9c150e1e5615497e53697ab440f07852e30a824f9676714da6fecf2b28e not found: ID does not exist" Oct 30 08:06:17.079876 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:17.079837 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz"] Oct 30 08:06:17.084919 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:17.084893 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ctldz"] Oct 30 08:06:17.774750 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:17.774715 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d729858d-5129-43e9-b6fe-129b8267791e" path="/var/lib/kubelet/pods/d729858d-5129-43e9-b6fe-129b8267791e/volumes" Oct 30 08:06:23.025347 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:06:23.025303 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 08:06:31.944010 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.943972 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp"] Oct 30 08:06:31.944890 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.944863 4951 cpu_manager.go:414] "containerMap: removing container" podUID="3933eee7-a582-4582-a7dc-23763490fad8" containerName="cleanup" Oct 30 08:06:31.945024 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.944891 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3933eee7-a582-4582-a7dc-23763490fad8" containerName="cleanup" Oct 30 08:06:31.945024 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.944908 4951 cpu_manager.go:414] "containerMap: removing container" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" Oct 30 08:06:31.945024 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.944916 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" Oct 30 08:06:31.945024 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.944934 4951 cpu_manager.go:414] "containerMap: removing container" podUID="705d9fa4-a31c-4fc9-b27b-48874576f32a" containerName="istio-proxy" Oct 30 08:06:31.945024 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.944942 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="705d9fa4-a31c-4fc9-b27b-48874576f32a" containerName="istio-proxy" Oct 30 08:06:31.945024 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.944959 4951 cpu_manager.go:414] "containerMap: removing container" podUID="0628b544-4ce0-4902-a5e8-5fef9194d1a2" containerName="maas-api" Oct 30 08:06:31.945024 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.944966 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0628b544-4ce0-4902-a5e8-5fef9194d1a2" containerName="maas-api" Oct 30 08:06:31.945024 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.944982 4951 cpu_manager.go:414] "containerMap: removing container" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" Oct 30 08:06:31.945024 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.944989 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" Oct 30 08:06:31.945024 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945007 4951 cpu_manager.go:414] "containerMap: removing container" podUID="f52156d3-44fc-4983-984c-8ac36b1cb942" containerName="istio-proxy" Oct 30 08:06:31.945024 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945016 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f52156d3-44fc-4983-984c-8ac36b1cb942" containerName="istio-proxy" Oct 30 08:06:31.945024 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945025 4951 cpu_manager.go:414] "containerMap: removing container" podUID="d3ce0d73-117c-4309-9f1d-71a0a6c53299" containerName="cleanup" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945034 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3ce0d73-117c-4309-9f1d-71a0a6c53299" containerName="cleanup" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945055 4951 cpu_manager.go:414] "containerMap: removing container" podUID="f52156d3-44fc-4983-984c-8ac36b1cb942" containerName="istio-proxy" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945062 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f52156d3-44fc-4983-984c-8ac36b1cb942" containerName="istio-proxy" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945080 4951 cpu_manager.go:414] "containerMap: removing container" podUID="d729858d-5129-43e9-b6fe-129b8267791e" containerName="discovery" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945087 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d729858d-5129-43e9-b6fe-129b8267791e" containerName="discovery" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945099 4951 cpu_manager.go:414] "containerMap: removing container" podUID="e684243f-7d08-4ae2-b946-7cea93656322" containerName="istio-proxy" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945107 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e684243f-7d08-4ae2-b946-7cea93656322" containerName="istio-proxy" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945116 4951 cpu_manager.go:414] "containerMap: removing container" podUID="e1304cf8-fc22-48db-acce-17514c25cf9d" containerName="istio-proxy" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945122 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1304cf8-fc22-48db-acce-17514c25cf9d" containerName="istio-proxy" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945134 4951 cpu_manager.go:414] "containerMap: removing container" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945141 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945159 4951 cpu_manager.go:414] "containerMap: removing container" podUID="9c290e9a-af71-48e8-99c6-43056987a08d" containerName="payload-pre-processing" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945166 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c290e9a-af71-48e8-99c6-43056987a08d" containerName="payload-pre-processing" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945180 4951 cpu_manager.go:414] "containerMap: removing container" podUID="50da1f49-e013-4e3e-a139-99d4ae6d668f" containerName="payload-pre-processing" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945187 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50da1f49-e013-4e3e-a139-99d4ae6d668f" containerName="payload-pre-processing" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945200 4951 cpu_manager.go:414] "containerMap: removing container" podUID="bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4" containerName="istio-proxy" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945207 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4" containerName="istio-proxy" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945224 4951 cpu_manager.go:414] "containerMap: removing container" podUID="bb29167c-8d9b-4938-afb5-0c974e7e34e7" containerName="maas-api" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945231 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb29167c-8d9b-4938-afb5-0c974e7e34e7" containerName="maas-api" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945243 4951 cpu_manager.go:414] "containerMap: removing container" podUID="cd68fbab-46a8-4bec-8c75-e6d3e07bf691" containerName="payload-processing" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945251 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd68fbab-46a8-4bec-8c75-e6d3e07bf691" containerName="payload-processing" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945266 4951 cpu_manager.go:414] "containerMap: removing container" podUID="bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4" containerName="istio-proxy" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945274 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4" containerName="istio-proxy" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945299 4951 cpu_manager.go:414] "containerMap: removing container" podUID="f014cb28-67ea-4356-a8a2-846720aaaa6d" containerName="maas-api" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945306 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f014cb28-67ea-4356-a8a2-846720aaaa6d" containerName="maas-api" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945322 4951 cpu_manager.go:414] "containerMap: removing container" podUID="ffb358d7-f1b3-4e53-b19e-85031c692e2b" containerName="maas-api" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945329 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffb358d7-f1b3-4e53-b19e-85031c692e2b" containerName="maas-api" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945341 4951 cpu_manager.go:414] "containerMap: removing container" podUID="82220bd1-27c2-447d-967d-80032aa1480d" containerName="cleanup" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945349 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82220bd1-27c2-447d-967d-80032aa1480d" containerName="cleanup" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945361 4951 cpu_manager.go:414] "containerMap: removing container" podUID="399020ed-d53c-41b2-8b3d-298647396c87" containerName="payload-processing" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945369 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="399020ed-d53c-41b2-8b3d-298647396c87" containerName="payload-processing" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945384 4951 cpu_manager.go:414] "containerMap: removing container" podUID="e684243f-7d08-4ae2-b946-7cea93656322" containerName="istio-proxy" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945392 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e684243f-7d08-4ae2-b946-7cea93656322" containerName="istio-proxy" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945554 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="705d9fa4-a31c-4fc9-b27b-48874576f32a" containerName="istio-proxy" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945589 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="0628b544-4ce0-4902-a5e8-5fef9194d1a2" containerName="maas-api" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945616 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="61d8d83a-32ea-43b0-9aa7-b70bff2217b8" containerName="main" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945627 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="f52156d3-44fc-4983-984c-8ac36b1cb942" containerName="istio-proxy" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945636 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3ce0d73-117c-4309-9f1d-71a0a6c53299" containerName="cleanup" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945646 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="f52156d3-44fc-4983-984c-8ac36b1cb942" containerName="istio-proxy" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945656 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="d729858d-5129-43e9-b6fe-129b8267791e" containerName="discovery" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945665 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="e684243f-7d08-4ae2-b946-7cea93656322" containerName="istio-proxy" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945674 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="fab18d8f-7bf2-44eb-a099-fd2a60990d0a" containerName="main" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945685 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c290e9a-af71-48e8-99c6-43056987a08d" containerName="payload-pre-processing" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945694 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="50da1f49-e013-4e3e-a139-99d4ae6d668f" containerName="payload-pre-processing" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945704 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4" containerName="istio-proxy" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945714 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb29167c-8d9b-4938-afb5-0c974e7e34e7" containerName="maas-api" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945723 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd68fbab-46a8-4bec-8c75-e6d3e07bf691" containerName="payload-processing" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945733 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf4c56f9-5d9c-40a0-8cd4-490d8459e4c4" containerName="istio-proxy" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945745 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c9697a-7dea-4662-8da4-c2618945d160" containerName="istio-proxy" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945755 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="f014cb28-67ea-4356-a8a2-846720aaaa6d" containerName="maas-api" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945766 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffb358d7-f1b3-4e53-b19e-85031c692e2b" containerName="maas-api" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945776 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="82220bd1-27c2-447d-967d-80032aa1480d" containerName="cleanup" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945784 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="399020ed-d53c-41b2-8b3d-298647396c87" containerName="payload-processing" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945795 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="3933eee7-a582-4582-a7dc-23763490fad8" containerName="cleanup" Oct 30 08:06:31.945784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.945803 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="d680a08d-af67-455d-87bc-9fa6a26ad37a" containerName="main" Oct 30 08:06:31.951314 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.951290 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:31.972507 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.972488 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-989104-openshift-default-dockercfg-59vg7\"" Oct 30 08:06:31.996141 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:31.996103 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp"] Oct 30 08:06:32.040335 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:32.040308 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:32.040484 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:32.040419 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:32.040484 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:32.040471 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:32.040605 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:32.040491 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:32.040605 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:32.040530 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:32.040707 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:32.040638 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:32.040748 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:32.040718 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:32.040795 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:32.040754 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2vnf\" (UniqueName: \"kubernetes.io/projected/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-kube-api-access-n2vnf\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:32.040795 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:32.040782 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:32.142310 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:32.142274 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:32.142530 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:32.142328 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:32.142530 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:32.142351 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:32.142530 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:32.142387 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:32.142530 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:32.142404 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:32.142530 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:32.142453 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:32.142530 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:32.142489 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2vnf\" (UniqueName: \"kubernetes.io/projected/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-kube-api-access-n2vnf\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:32.142530 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:32.142517 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:32.143034 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:32.142602 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:32.143034 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:32.142866 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:32.143034 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:32.142953 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:32.143200 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:32.143147 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:32.143200 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:32.143159 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:32.143305 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:32.143250 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:32.144750 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:32.144725 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:32.144983 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:32.144964 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:32.183643 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:32.183567 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:32.197483 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:32.197419 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2vnf\" (UniqueName: \"kubernetes.io/projected/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-kube-api-access-n2vnf\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:32.270430 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:32.270389 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:32.623220 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:32.623178 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 08:06:32.623353 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:32.623252 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 08:06:32.623353 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:32.623284 4951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 08:06:32.624733 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:32.624701 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp"] Oct 30 08:06:33.153949 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:33.153904 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" event={"ID":"8f3ce93c-f668-4afc-b693-bfb8f0ad8428","Type":"ContainerStarted","Data":"c499e7056d83c69dc55be950a0c3186877ebcbf7610acfd23ac6473c84914edd"} Oct 30 08:06:33.154376 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:33.153969 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" event={"ID":"8f3ce93c-f668-4afc-b693-bfb8f0ad8428","Type":"ContainerStarted","Data":"d43e004b2bd7bca470cea69c073cea68fd83d6d940aecd50703d9826c744ea24"} Oct 30 08:06:33.241530 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:33.241454 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" podStartSLOduration=2.241432626 podStartE2EDuration="2.241432626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:06:33.240856105 +0000 UTC m=+4199.583904072" watchObservedRunningTime="2026-10-30 08:06:33.241432626 +0000 UTC m=+4199.584480593" Oct 30 08:06:33.271163 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:33.271136 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:33.272868 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:33.272842 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Oct 30 08:06:33.273025 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:33.272892 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" podUID="8f3ce93c-f668-4afc-b693-bfb8f0ad8428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Oct 30 08:06:34.271415 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:34.271366 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Oct 30 08:06:34.271835 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:34.271469 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" podUID="8f3ce93c-f668-4afc-b693-bfb8f0ad8428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Oct 30 08:06:34.794220 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:34.794071 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/3.log" Oct 30 08:06:34.802831 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:34.802796 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 08:06:34.814719 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:34.814694 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/3.log" Oct 30 08:06:34.820933 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:34.820908 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tlbb_bd315a45-3a31-44f2-9bf1-6d43df8e0c44/ovn-acl-logging/0.log" Oct 30 08:06:35.271567 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:35.271535 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Oct 30 08:06:35.272007 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:35.271617 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" podUID="8f3ce93c-f668-4afc-b693-bfb8f0ad8428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Oct 30 08:06:36.271401 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:36.271363 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Oct 30 08:06:36.271610 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:36.271463 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" podUID="8f3ce93c-f668-4afc-b693-bfb8f0ad8428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Oct 30 08:06:36.541615 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:36.541502 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7602-0da41a"] Oct 30 08:06:36.542984 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:36.542954 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="e684243f-7d08-4ae2-b946-7cea93656322" containerName="istio-proxy" Oct 30 08:06:36.551252 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:36.551230 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-0da41a" Oct 30 08:06:36.566886 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:36.566862 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7602-0da41a"] Oct 30 08:06:36.575220 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:36.575193 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5z8jp\"" Oct 30 08:06:36.618950 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:36.618920 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb5jj\" (UniqueName: \"kubernetes.io/projected/3e881064-38e1-4be3-9ea1-6fcb6909e74a-kube-api-access-lb5jj\") pod \"test-curl-7602-0da41a\" (UID: \"3e881064-38e1-4be3-9ea1-6fcb6909e74a\") " pod="opendatahub/test-curl-7602-0da41a" Oct 30 08:06:36.720379 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:36.720342 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb5jj\" (UniqueName: \"kubernetes.io/projected/3e881064-38e1-4be3-9ea1-6fcb6909e74a-kube-api-access-lb5jj\") pod \"test-curl-7602-0da41a\" (UID: \"3e881064-38e1-4be3-9ea1-6fcb6909e74a\") " pod="opendatahub/test-curl-7602-0da41a" Oct 30 08:06:36.749323 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:36.749292 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb5jj\" (UniqueName: \"kubernetes.io/projected/3e881064-38e1-4be3-9ea1-6fcb6909e74a-kube-api-access-lb5jj\") pod \"test-curl-7602-0da41a\" (UID: \"3e881064-38e1-4be3-9ea1-6fcb6909e74a\") " pod="opendatahub/test-curl-7602-0da41a" Oct 30 08:06:36.867112 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:36.867019 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-0da41a" Oct 30 08:06:37.190375 ip-10-0-132-173 kubenswrapper[4951]: W1030 08:06:37.190336 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e881064_38e1_4be3_9ea1_6fcb6909e74a.slice/crio-0a04ae0352acf530cf991afe549d69831f1a356e4e4a6c8b20da0a20d27fb3e2 WatchSource:0}: Error finding container 0a04ae0352acf530cf991afe549d69831f1a356e4e4a6c8b20da0a20d27fb3e2: Status 404 returned error can't find the container with id 0a04ae0352acf530cf991afe549d69831f1a356e4e4a6c8b20da0a20d27fb3e2 Oct 30 08:06:37.190754 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:37.190716 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7602-0da41a"] Oct 30 08:06:37.270904 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:37.270875 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Oct 30 08:06:37.271124 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:37.270921 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" podUID="8f3ce93c-f668-4afc-b693-bfb8f0ad8428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Oct 30 08:06:38.188344 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:38.188306 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7602-0da41a" event={"ID":"3e881064-38e1-4be3-9ea1-6fcb6909e74a","Type":"ContainerStarted","Data":"2abefd920dc3d9e555351b2b7e3b557f841760bbb7fcea50ad23c93713a241a9"} Oct 30 08:06:38.188873 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:38.188351 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7602-0da41a" event={"ID":"3e881064-38e1-4be3-9ea1-6fcb6909e74a","Type":"ContainerStarted","Data":"0a04ae0352acf530cf991afe549d69831f1a356e4e4a6c8b20da0a20d27fb3e2"} Oct 30 08:06:38.250696 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:38.250626 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7602-0da41a" podStartSLOduration=2.250606257 podStartE2EDuration="2.250606257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:06:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:06:38.249941279 +0000 UTC m=+4204.592989244" watchObservedRunningTime="2026-10-30 08:06:38.250606257 +0000 UTC m=+4204.593654224" Oct 30 08:06:38.271585 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:38.271538 4951 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Oct 30 08:06:38.271775 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:38.271610 4951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" podUID="8f3ce93c-f668-4afc-b693-bfb8f0ad8428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Oct 30 08:06:39.104617 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:39.104564 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7602-0da41a"] Oct 30 08:06:39.110161 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:39.110131 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7602-0da41a"] Oct 30 08:06:39.266081 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:39.266049 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7602-92594c"] Oct 30 08:06:39.266767 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:39.266746 4951 cpu_manager.go:414] "containerMap: removing container" podUID="3e881064-38e1-4be3-9ea1-6fcb6909e74a" containerName="test-curl-7602-0da41a" Oct 30 08:06:39.266767 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:39.266765 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e881064-38e1-4be3-9ea1-6fcb6909e74a" containerName="test-curl-7602-0da41a" Oct 30 08:06:39.266956 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:39.266873 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e881064-38e1-4be3-9ea1-6fcb6909e74a" containerName="test-curl-7602-0da41a" Oct 30 08:06:39.271655 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:39.271635 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-92594c" Oct 30 08:06:39.276427 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:39.276401 4951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:39.276731 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:39.276708 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:39.277467 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:39.277448 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:39.286624 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:39.286600 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7602-92594c"] Oct 30 08:06:39.382869 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:39.382771 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9swjs\" (UniqueName: \"kubernetes.io/projected/334b282b-4bb9-4ebf-a675-b6940059cdd6-kube-api-access-9swjs\") pod \"test-curl-7602-92594c\" (UID: \"334b282b-4bb9-4ebf-a675-b6940059cdd6\") " pod="opendatahub/test-curl-7602-92594c" Oct 30 08:06:39.484595 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:39.484542 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9swjs\" (UniqueName: \"kubernetes.io/projected/334b282b-4bb9-4ebf-a675-b6940059cdd6-kube-api-access-9swjs\") pod \"test-curl-7602-92594c\" (UID: \"334b282b-4bb9-4ebf-a675-b6940059cdd6\") " pod="opendatahub/test-curl-7602-92594c" Oct 30 08:06:39.509038 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:39.509006 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9swjs\" (UniqueName: \"kubernetes.io/projected/334b282b-4bb9-4ebf-a675-b6940059cdd6-kube-api-access-9swjs\") pod \"test-curl-7602-92594c\" (UID: \"334b282b-4bb9-4ebf-a675-b6940059cdd6\") " pod="opendatahub/test-curl-7602-92594c" Oct 30 08:06:39.584506 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:39.584471 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-92594c" Oct 30 08:06:39.900276 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:39.900242 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7602-92594c"] Oct 30 08:06:39.900444 ip-10-0-132-173 kubenswrapper[4951]: W1030 08:06:39.900359 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod334b282b_4bb9_4ebf_a675_b6940059cdd6.slice/crio-139236388f45887ad6024fddaa9ae0e715d671010b2d164925dd0f1874783df9 WatchSource:0}: Error finding container 139236388f45887ad6024fddaa9ae0e715d671010b2d164925dd0f1874783df9: Status 404 returned error can't find the container with id 139236388f45887ad6024fddaa9ae0e715d671010b2d164925dd0f1874783df9 Oct 30 08:06:40.216291 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:40.216209 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7602-92594c" event={"ID":"334b282b-4bb9-4ebf-a675-b6940059cdd6","Type":"ContainerStarted","Data":"2b91f09413a0584baa5b30ffc8ea838ae1504dd153dba1aee2dea36a7173a7df"} Oct 30 08:06:40.216291 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:40.216255 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7602-92594c" event={"ID":"334b282b-4bb9-4ebf-a675-b6940059cdd6","Type":"ContainerStarted","Data":"139236388f45887ad6024fddaa9ae0e715d671010b2d164925dd0f1874783df9"} Oct 30 08:06:40.216662 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:40.216640 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7602-0da41a" podUID="3e881064-38e1-4be3-9ea1-6fcb6909e74a" containerName="test-curl-7602-0da41a" containerID="cri-o://2abefd920dc3d9e555351b2b7e3b557f841760bbb7fcea50ad23c93713a241a9" gracePeriod=2 Oct 30 08:06:40.258681 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:40.258623 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7602-92594c" podStartSLOduration=1.258610579 podStartE2EDuration="1.258610579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:06:40.258385743 +0000 UTC m=+4206.601433737" watchObservedRunningTime="2026-10-30 08:06:40.258610579 +0000 UTC m=+4206.601658544" Oct 30 08:06:41.122963 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:41.122929 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7602-92594c"] Oct 30 08:06:41.129607 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:41.129559 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7602-92594c"] Oct 30 08:06:42.235494 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:42.235461 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7602-92594c" podUID="334b282b-4bb9-4ebf-a675-b6940059cdd6" containerName="test-curl-7602-92594c" containerID="cri-o://2b91f09413a0584baa5b30ffc8ea838ae1504dd153dba1aee2dea36a7173a7df" gracePeriod=2 Oct 30 08:06:42.436805 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:42.436772 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-0da41a" Oct 30 08:06:42.450558 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:42.450524 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp"] Oct 30 08:06:42.450785 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:42.450758 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" podUID="8f3ce93c-f668-4afc-b693-bfb8f0ad8428" containerName="istio-proxy" containerID="cri-o://c499e7056d83c69dc55be950a0c3186877ebcbf7610acfd23ac6473c84914edd" gracePeriod=30 Oct 30 08:06:42.522446 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:42.522416 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e881064-38e1-4be3-9ea1-6fcb6909e74a-kube-api-access-lb5jj\" (UniqueName: \"kubernetes.io/projected/3e881064-38e1-4be3-9ea1-6fcb6909e74a-kube-api-access-lb5jj\") pod \"3e881064-38e1-4be3-9ea1-6fcb6909e74a\" (UID: \"3e881064-38e1-4be3-9ea1-6fcb6909e74a\") " Oct 30 08:06:42.524477 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:42.524451 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e881064-38e1-4be3-9ea1-6fcb6909e74a-kube-api-access-lb5jj" pod "3e881064-38e1-4be3-9ea1-6fcb6909e74a" (UID: "3e881064-38e1-4be3-9ea1-6fcb6909e74a"). InnerVolumeSpecName "kube-api-access-lb5jj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:06:42.624299 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:42.624255 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb5jj\" (UniqueName: \"kubernetes.io/projected/3e881064-38e1-4be3-9ea1-6fcb6909e74a-kube-api-access-lb5jj\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:06:43.244290 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:43.244247 4951 generic.go:356] "Generic (PLEG): container finished" podID="3e881064-38e1-4be3-9ea1-6fcb6909e74a" containerID="2abefd920dc3d9e555351b2b7e3b557f841760bbb7fcea50ad23c93713a241a9" exitCode=137 Oct 30 08:06:43.244835 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:43.244298 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-0da41a" Oct 30 08:06:43.244835 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:43.244339 4951 scope.go:122] "RemoveContainer" containerID="2abefd920dc3d9e555351b2b7e3b557f841760bbb7fcea50ad23c93713a241a9" Oct 30 08:06:43.258832 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:43.258806 4951 scope.go:122] "RemoveContainer" containerID="2abefd920dc3d9e555351b2b7e3b557f841760bbb7fcea50ad23c93713a241a9" Oct 30 08:06:43.259299 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:06:43.259272 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2abefd920dc3d9e555351b2b7e3b557f841760bbb7fcea50ad23c93713a241a9\": container with ID starting with 2abefd920dc3d9e555351b2b7e3b557f841760bbb7fcea50ad23c93713a241a9 not found: ID does not exist" containerID="2abefd920dc3d9e555351b2b7e3b557f841760bbb7fcea50ad23c93713a241a9" Oct 30 08:06:43.259389 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:43.259312 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2abefd920dc3d9e555351b2b7e3b557f841760bbb7fcea50ad23c93713a241a9"} err="failed to get container status \"2abefd920dc3d9e555351b2b7e3b557f841760bbb7fcea50ad23c93713a241a9\": rpc error: code = NotFound desc = could not find container \"2abefd920dc3d9e555351b2b7e3b557f841760bbb7fcea50ad23c93713a241a9\": container with ID starting with 2abefd920dc3d9e555351b2b7e3b557f841760bbb7fcea50ad23c93713a241a9 not found: ID does not exist" Oct 30 08:06:43.271015 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:06:43.270980 4951 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7602-0da41a\" is forbidden: User \"system:node:ip-10-0-132-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-173.ec2.internal' and this object" podUID="3e881064-38e1-4be3-9ea1-6fcb6909e74a" pod="opendatahub/test-curl-7602-0da41a" Oct 30 08:06:43.782690 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:43.782654 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e881064-38e1-4be3-9ea1-6fcb6909e74a" path="/var/lib/kubelet/pods/3e881064-38e1-4be3-9ea1-6fcb6909e74a/volumes" Oct 30 08:06:43.796341 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:06:43.796312 4951 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7602-0da41a\" is forbidden: User \"system:node:ip-10-0-132-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-173.ec2.internal' and this object" podUID="3e881064-38e1-4be3-9ea1-6fcb6909e74a" pod="opendatahub/test-curl-7602-0da41a" Oct 30 08:06:44.461252 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:44.461221 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-92594c" Oct 30 08:06:44.549930 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:44.549890 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/334b282b-4bb9-4ebf-a675-b6940059cdd6-kube-api-access-9swjs\" (UniqueName: \"kubernetes.io/projected/334b282b-4bb9-4ebf-a675-b6940059cdd6-kube-api-access-9swjs\") pod \"334b282b-4bb9-4ebf-a675-b6940059cdd6\" (UID: \"334b282b-4bb9-4ebf-a675-b6940059cdd6\") " Oct 30 08:06:44.552219 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:44.552185 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334b282b-4bb9-4ebf-a675-b6940059cdd6-kube-api-access-9swjs" pod "334b282b-4bb9-4ebf-a675-b6940059cdd6" (UID: "334b282b-4bb9-4ebf-a675-b6940059cdd6"). InnerVolumeSpecName "kube-api-access-9swjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:06:44.651209 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:44.651170 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9swjs\" (UniqueName: \"kubernetes.io/projected/334b282b-4bb9-4ebf-a675-b6940059cdd6-kube-api-access-9swjs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:06:45.263061 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:45.263017 4951 generic.go:356] "Generic (PLEG): container finished" podID="334b282b-4bb9-4ebf-a675-b6940059cdd6" containerID="2b91f09413a0584baa5b30ffc8ea838ae1504dd153dba1aee2dea36a7173a7df" exitCode=137 Oct 30 08:06:45.263283 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:45.263072 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-92594c" Oct 30 08:06:45.263283 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:45.263106 4951 scope.go:122] "RemoveContainer" containerID="2b91f09413a0584baa5b30ffc8ea838ae1504dd153dba1aee2dea36a7173a7df" Oct 30 08:06:45.274452 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:45.274258 4951 scope.go:122] "RemoveContainer" containerID="2b91f09413a0584baa5b30ffc8ea838ae1504dd153dba1aee2dea36a7173a7df" Oct 30 08:06:45.274549 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:06:45.274530 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b91f09413a0584baa5b30ffc8ea838ae1504dd153dba1aee2dea36a7173a7df\": container with ID starting with 2b91f09413a0584baa5b30ffc8ea838ae1504dd153dba1aee2dea36a7173a7df not found: ID does not exist" containerID="2b91f09413a0584baa5b30ffc8ea838ae1504dd153dba1aee2dea36a7173a7df" Oct 30 08:06:45.274624 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:45.274559 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b91f09413a0584baa5b30ffc8ea838ae1504dd153dba1aee2dea36a7173a7df"} err="failed to get container status \"2b91f09413a0584baa5b30ffc8ea838ae1504dd153dba1aee2dea36a7173a7df\": rpc error: code = NotFound desc = could not find container \"2b91f09413a0584baa5b30ffc8ea838ae1504dd153dba1aee2dea36a7173a7df\": container with ID starting with 2b91f09413a0584baa5b30ffc8ea838ae1504dd153dba1aee2dea36a7173a7df not found: ID does not exist" Oct 30 08:06:45.288261 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:06:45.288235 4951 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7602-92594c\" is forbidden: User \"system:node:ip-10-0-132-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-173.ec2.internal' and this object" podUID="334b282b-4bb9-4ebf-a675-b6940059cdd6" pod="opendatahub/test-curl-7602-92594c" Oct 30 08:06:45.774055 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:45.774020 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="334b282b-4bb9-4ebf-a675-b6940059cdd6" path="/var/lib/kubelet/pods/334b282b-4bb9-4ebf-a675-b6940059cdd6/volumes" Oct 30 08:06:47.739816 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:47.739786 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:47.797807 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:47.797776 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-workload-socket\") pod \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " Oct 30 08:06:47.797980 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:47.797902 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istio-data\") pod \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " Oct 30 08:06:47.798044 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:47.798003 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-workload-certs\") pod \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " Oct 30 08:06:47.798094 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:47.798080 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istio-token\" (UniqueName: \"kubernetes.io/projected/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istio-token\") pod \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " Oct 30 08:06:47.798144 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:47.798083 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-workload-socket" pod "8f3ce93c-f668-4afc-b693-bfb8f0ad8428" (UID: "8f3ce93c-f668-4afc-b693-bfb8f0ad8428"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:47.798204 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:47.798171 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istio-envoy\") pod \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " Oct 30 08:06:47.798258 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:47.798227 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istio-podinfo\") pod \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " Oct 30 08:06:47.798311 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:47.798276 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-kube-api-access-n2vnf\" (UniqueName: \"kubernetes.io/projected/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-kube-api-access-n2vnf\") pod \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " Oct 30 08:06:47.798360 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:47.798319 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-credential-socket\") pod \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " Oct 30 08:06:47.798360 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:47.798347 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istiod-ca-cert\") pod \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\" (UID: \"8f3ce93c-f668-4afc-b693-bfb8f0ad8428\") " Oct 30 08:06:47.798360 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:47.798347 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istio-data" pod "8f3ce93c-f668-4afc-b693-bfb8f0ad8428" (UID: "8f3ce93c-f668-4afc-b693-bfb8f0ad8428"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:47.798606 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:47.798462 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-workload-certs" pod "8f3ce93c-f668-4afc-b693-bfb8f0ad8428" (UID: "8f3ce93c-f668-4afc-b693-bfb8f0ad8428"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:47.799208 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:47.798871 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:06:47.799208 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:47.798896 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:06:47.799208 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:47.798914 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:06:47.799208 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:47.799159 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-credential-socket" pod "8f3ce93c-f668-4afc-b693-bfb8f0ad8428" (UID: "8f3ce93c-f668-4afc-b693-bfb8f0ad8428"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:47.802831 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:47.802805 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istiod-ca-cert" pod "8f3ce93c-f668-4afc-b693-bfb8f0ad8428" (UID: "8f3ce93c-f668-4afc-b693-bfb8f0ad8428"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:06:47.805312 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:47.804343 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istio-podinfo" pod "8f3ce93c-f668-4afc-b693-bfb8f0ad8428" (UID: "8f3ce93c-f668-4afc-b693-bfb8f0ad8428"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:06:47.805312 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:47.804384 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istio-token" pod "8f3ce93c-f668-4afc-b693-bfb8f0ad8428" (UID: "8f3ce93c-f668-4afc-b693-bfb8f0ad8428"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:06:47.805312 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:47.804486 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istio-envoy" pod "8f3ce93c-f668-4afc-b693-bfb8f0ad8428" (UID: "8f3ce93c-f668-4afc-b693-bfb8f0ad8428"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:47.805312 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:47.805254 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-kube-api-access-n2vnf" pod "8f3ce93c-f668-4afc-b693-bfb8f0ad8428" (UID: "8f3ce93c-f668-4afc-b693-bfb8f0ad8428"). InnerVolumeSpecName "kube-api-access-n2vnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:06:47.899933 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:47.899898 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:06:47.899933 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:47.899927 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:06:47.899933 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:47.899937 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2vnf\" (UniqueName: \"kubernetes.io/projected/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-kube-api-access-n2vnf\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:06:47.900186 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:47.899948 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:06:47.900186 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:47.899957 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:06:47.900186 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:47.899965 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f3ce93c-f668-4afc-b693-bfb8f0ad8428-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:06:48.309202 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:48.309031 4951 generic.go:356] "Generic (PLEG): container finished" podID="8f3ce93c-f668-4afc-b693-bfb8f0ad8428" containerID="c499e7056d83c69dc55be950a0c3186877ebcbf7610acfd23ac6473c84914edd" exitCode=0 Oct 30 08:06:48.309202 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:48.309111 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" Oct 30 08:06:48.309202 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:48.309106 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" event={"ID":"8f3ce93c-f668-4afc-b693-bfb8f0ad8428","Type":"ContainerDied","Data":"c499e7056d83c69dc55be950a0c3186877ebcbf7610acfd23ac6473c84914edd"} Oct 30 08:06:48.309535 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:48.309274 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp" event={"ID":"8f3ce93c-f668-4afc-b693-bfb8f0ad8428","Type":"ContainerDied","Data":"d43e004b2bd7bca470cea69c073cea68fd83d6d940aecd50703d9826c744ea24"} Oct 30 08:06:48.309535 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:48.309302 4951 scope.go:122] "RemoveContainer" containerID="c499e7056d83c69dc55be950a0c3186877ebcbf7610acfd23ac6473c84914edd" Oct 30 08:06:48.322381 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:48.322359 4951 scope.go:122] "RemoveContainer" containerID="c499e7056d83c69dc55be950a0c3186877ebcbf7610acfd23ac6473c84914edd" Oct 30 08:06:48.322704 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:06:48.322680 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c499e7056d83c69dc55be950a0c3186877ebcbf7610acfd23ac6473c84914edd\": container with ID starting with c499e7056d83c69dc55be950a0c3186877ebcbf7610acfd23ac6473c84914edd not found: ID does not exist" containerID="c499e7056d83c69dc55be950a0c3186877ebcbf7610acfd23ac6473c84914edd" Oct 30 08:06:48.322818 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:48.322714 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c499e7056d83c69dc55be950a0c3186877ebcbf7610acfd23ac6473c84914edd"} err="failed to get container status \"c499e7056d83c69dc55be950a0c3186877ebcbf7610acfd23ac6473c84914edd\": rpc error: code = NotFound desc = could not find container \"c499e7056d83c69dc55be950a0c3186877ebcbf7610acfd23ac6473c84914edd\": container with ID starting with c499e7056d83c69dc55be950a0c3186877ebcbf7610acfd23ac6473c84914edd not found: ID does not exist" Oct 30 08:06:48.377103 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:48.377069 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp"] Oct 30 08:06:48.389710 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:48.389672 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-r29bp"] Oct 30 08:06:49.773128 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:06:49.773095 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f3ce93c-f668-4afc-b693-bfb8f0ad8428" path="/var/lib/kubelet/pods/8f3ce93c-f668-4afc-b693-bfb8f0ad8428/volumes" Oct 30 08:06:53.026911 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:06:53.026872 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 08:07:23.026319 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:07:23.026279 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 08:07:29.452513 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:29.452476 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1e0880-79484bb6bd-5vfjq"] Oct 30 08:07:29.484213 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:29.484182 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1e0880-6946fccf5f-2rcnf"] Oct 30 08:07:33.412053 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:33.412014 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk"] Oct 30 08:07:33.609474 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:33.609417 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk" podUID="92a44e19-26dc-46b2-8a2b-5d5a33941087" containerName="maas-api" containerID="cri-o://754c26b3e50f88bb715e21fa391e5454996543d40f83f00364b6834c26ba501b" gracePeriod=30 Oct 30 08:07:33.952642 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:33.952609 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk" Oct 30 08:07:34.010037 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:34.009999 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92a44e19-26dc-46b2-8a2b-5d5a33941087-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92a44e19-26dc-46b2-8a2b-5d5a33941087-maas-api-tls\") pod \"92a44e19-26dc-46b2-8a2b-5d5a33941087\" (UID: \"92a44e19-26dc-46b2-8a2b-5d5a33941087\") " Oct 30 08:07:34.010238 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:34.010060 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92a44e19-26dc-46b2-8a2b-5d5a33941087-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92a44e19-26dc-46b2-8a2b-5d5a33941087-openshift-service-ca\") pod \"92a44e19-26dc-46b2-8a2b-5d5a33941087\" (UID: \"92a44e19-26dc-46b2-8a2b-5d5a33941087\") " Oct 30 08:07:34.010238 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:34.010202 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a44e19-26dc-46b2-8a2b-5d5a33941087-kube-api-access-c5hbb\" (UniqueName: \"kubernetes.io/projected/92a44e19-26dc-46b2-8a2b-5d5a33941087-kube-api-access-c5hbb\") pod \"92a44e19-26dc-46b2-8a2b-5d5a33941087\" (UID: \"92a44e19-26dc-46b2-8a2b-5d5a33941087\") " Oct 30 08:07:34.010331 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:34.010276 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92a44e19-26dc-46b2-8a2b-5d5a33941087-metrics-certs\" (UniqueName: \"kubernetes.io/secret/92a44e19-26dc-46b2-8a2b-5d5a33941087-metrics-certs\") pod \"92a44e19-26dc-46b2-8a2b-5d5a33941087\" (UID: \"92a44e19-26dc-46b2-8a2b-5d5a33941087\") " Oct 30 08:07:34.010431 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:34.010399 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92a44e19-26dc-46b2-8a2b-5d5a33941087-openshift-service-ca" pod "92a44e19-26dc-46b2-8a2b-5d5a33941087" (UID: "92a44e19-26dc-46b2-8a2b-5d5a33941087"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:07:34.010861 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:34.010825 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92a44e19-26dc-46b2-8a2b-5d5a33941087-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:07:34.012526 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:34.012494 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92a44e19-26dc-46b2-8a2b-5d5a33941087-maas-api-tls" pod "92a44e19-26dc-46b2-8a2b-5d5a33941087" (UID: "92a44e19-26dc-46b2-8a2b-5d5a33941087"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:07:34.012869 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:34.012841 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92a44e19-26dc-46b2-8a2b-5d5a33941087-metrics-certs" pod "92a44e19-26dc-46b2-8a2b-5d5a33941087" (UID: "92a44e19-26dc-46b2-8a2b-5d5a33941087"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:07:34.013123 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:34.013096 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a44e19-26dc-46b2-8a2b-5d5a33941087-kube-api-access-c5hbb" pod "92a44e19-26dc-46b2-8a2b-5d5a33941087" (UID: "92a44e19-26dc-46b2-8a2b-5d5a33941087"). InnerVolumeSpecName "kube-api-access-c5hbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:07:34.112377 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:34.112337 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5hbb\" (UniqueName: \"kubernetes.io/projected/92a44e19-26dc-46b2-8a2b-5d5a33941087-kube-api-access-c5hbb\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:07:34.112377 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:34.112379 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92a44e19-26dc-46b2-8a2b-5d5a33941087-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:07:34.112616 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:34.112394 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92a44e19-26dc-46b2-8a2b-5d5a33941087-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:07:34.453173 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:34.453067 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-1e0880-79484bb6bd-5vfjq" podUID="33977737-ec70-47e5-a689-4c471c77c42e" containerName="payload-processing" containerID="cri-o://e0cd5e62032638d8f0203a10f7dccac3d09d50a29ee671b3f5abc055bab6b1d9" gracePeriod=25 Oct 30 08:07:34.484692 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:34.484636 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1e0880-6946fccf5f-2rcnf" podUID="64ba04d1-bbfe-4ed3-b0b2-55d554a025fb" containerName="payload-pre-processing" containerID="cri-o://c7c773054675033d5f0b1cbd172417a233d38b94b95b6560c349a423282a997b" gracePeriod=25 Oct 30 08:07:34.618208 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:34.618168 4951 generic.go:356] "Generic (PLEG): container finished" podID="92a44e19-26dc-46b2-8a2b-5d5a33941087" containerID="754c26b3e50f88bb715e21fa391e5454996543d40f83f00364b6834c26ba501b" exitCode=0 Oct 30 08:07:34.618436 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:34.618333 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk" event={"ID":"92a44e19-26dc-46b2-8a2b-5d5a33941087","Type":"ContainerDied","Data":"754c26b3e50f88bb715e21fa391e5454996543d40f83f00364b6834c26ba501b"} Oct 30 08:07:34.618436 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:34.618376 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk" Oct 30 08:07:34.618436 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:34.618419 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk" event={"ID":"92a44e19-26dc-46b2-8a2b-5d5a33941087","Type":"ContainerDied","Data":"28cbe79cf483f13f4b332a48f31405386dfd0394222f1026b52cbb62038a891c"} Oct 30 08:07:34.621585 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:34.618449 4951 scope.go:122] "RemoveContainer" containerID="754c26b3e50f88bb715e21fa391e5454996543d40f83f00364b6834c26ba501b" Oct 30 08:07:34.635184 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:34.635161 4951 scope.go:122] "RemoveContainer" containerID="754c26b3e50f88bb715e21fa391e5454996543d40f83f00364b6834c26ba501b" Oct 30 08:07:34.635835 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:07:34.635801 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"754c26b3e50f88bb715e21fa391e5454996543d40f83f00364b6834c26ba501b\": container with ID starting with 754c26b3e50f88bb715e21fa391e5454996543d40f83f00364b6834c26ba501b not found: ID does not exist" containerID="754c26b3e50f88bb715e21fa391e5454996543d40f83f00364b6834c26ba501b" Oct 30 08:07:34.635952 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:34.635843 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"754c26b3e50f88bb715e21fa391e5454996543d40f83f00364b6834c26ba501b"} err="failed to get container status \"754c26b3e50f88bb715e21fa391e5454996543d40f83f00364b6834c26ba501b\": rpc error: code = NotFound desc = could not find container \"754c26b3e50f88bb715e21fa391e5454996543d40f83f00364b6834c26ba501b\": container with ID starting with 754c26b3e50f88bb715e21fa391e5454996543d40f83f00364b6834c26ba501b not found: ID does not exist" Oct 30 08:07:34.707435 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:34.707338 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk"] Oct 30 08:07:34.712894 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:34.712859 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb2e8f-84df56fbbd-jd5dk"] Oct 30 08:07:35.772433 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:35.772395 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92a44e19-26dc-46b2-8a2b-5d5a33941087" path="/var/lib/kubelet/pods/92a44e19-26dc-46b2-8a2b-5d5a33941087/volumes" Oct 30 08:07:39.008359 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:39.008323 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5"] Oct 30 08:07:39.008884 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:39.008542 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" podUID="9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3" containerName="istio-proxy" containerID="cri-o://947900cbb5ceaac48c9cf3052160be0a5312be6fea7b44fc83d757a6ddc0f1ff" gracePeriod=30 Oct 30 08:07:41.336371 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:41.336327 4951 patch_prober.go:29] interesting pod/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 08:07:41.336946 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:41.336408 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" podUID="9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 08:07:42.413435 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:42.413398 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc"] Oct 30 08:07:42.413958 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:42.413629 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" podUID="b08b669b-c0a7-462c-9b63-aabbb0254dc1" containerName="istio-proxy" containerID="cri-o://4d0bd6310830aebfaa316fa880e9568454ffeb3bc27c3f2b2a41f603cc76ad57" gracePeriod=30 Oct 30 08:07:44.014840 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.014795 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-1e0880-79484bb6bd-5vfjq" podUID="33977737-ec70-47e5-a689-4c471c77c42e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:07:44.015324 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.014804 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1e0880-6946fccf5f-2rcnf" podUID="64ba04d1-bbfe-4ed3-b0b2-55d554a025fb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:07:44.299857 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.299825 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 08:07:44.349004 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.348954 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istio-data\") pod \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " Oct 30 08:07:44.349236 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.349079 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istio-token\" (UniqueName: \"kubernetes.io/projected/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istio-token\") pod \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " Oct 30 08:07:44.349236 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.349149 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-credential-socket\") pod \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " Oct 30 08:07:44.349348 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.349240 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-workload-certs\") pod \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " Oct 30 08:07:44.349348 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.349311 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istiod-ca-cert\") pod \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " Oct 30 08:07:44.349348 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.349306 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istio-data" pod "9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3" (UID: "9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:07:44.349496 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.349345 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istio-envoy\") pod \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " Oct 30 08:07:44.349496 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.349370 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-credential-socket" pod "9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3" (UID: "9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:07:44.349496 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.349382 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istio-podinfo\") pod \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " Oct 30 08:07:44.349496 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.349418 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-kube-api-access-pmm9w\" (UniqueName: \"kubernetes.io/projected/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-kube-api-access-pmm9w\") pod \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " Oct 30 08:07:44.349496 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.349463 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-workload-socket\") pod \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\" (UID: \"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3\") " Oct 30 08:07:44.349758 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.349597 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-workload-certs" pod "9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3" (UID: "9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:07:44.349758 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.349679 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istiod-ca-cert" pod "9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3" (UID: "9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:07:44.350074 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.350045 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-workload-socket" pod "9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3" (UID: "9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:07:44.350494 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.350467 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:07:44.350623 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.350499 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:07:44.350623 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.350511 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:07:44.350623 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.350522 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:07:44.350623 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.350534 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:07:44.351517 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.351496 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istio-envoy" pod "9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3" (UID: "9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:07:44.351724 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.351704 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istio-token" pod "9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3" (UID: "9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:07:44.352011 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.351989 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-kube-api-access-pmm9w" pod "9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3" (UID: "9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3"). InnerVolumeSpecName "kube-api-access-pmm9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:07:44.352674 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.352648 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istio-podinfo" pod "9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3" (UID: "9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:07:44.452466 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.452421 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:07:44.452466 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.452464 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:07:44.452732 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.452477 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:07:44.452732 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.452492 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmm9w\" (UniqueName: \"kubernetes.io/projected/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3-kube-api-access-pmm9w\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:07:44.700954 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.700851 4951 generic.go:356] "Generic (PLEG): container finished" podID="9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3" containerID="947900cbb5ceaac48c9cf3052160be0a5312be6fea7b44fc83d757a6ddc0f1ff" exitCode=0 Oct 30 08:07:44.700954 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.700903 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" Oct 30 08:07:44.700954 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.700942 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" event={"ID":"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3","Type":"ContainerDied","Data":"947900cbb5ceaac48c9cf3052160be0a5312be6fea7b44fc83d757a6ddc0f1ff"} Oct 30 08:07:44.701247 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.701005 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5" event={"ID":"9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3","Type":"ContainerDied","Data":"b582f9ff8c31624554f64b79f6d0f851c2468eb102bc06c797694be7b282eb41"} Oct 30 08:07:44.701247 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.701029 4951 scope.go:122] "RemoveContainer" containerID="947900cbb5ceaac48c9cf3052160be0a5312be6fea7b44fc83d757a6ddc0f1ff" Oct 30 08:07:44.712856 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.712836 4951 scope.go:122] "RemoveContainer" containerID="5c6a8eec409b079ca0e33fab991eb8bbaae4ac44bce0f48415332c5b33d0dbc2" Oct 30 08:07:44.722460 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.722437 4951 scope.go:122] "RemoveContainer" containerID="947900cbb5ceaac48c9cf3052160be0a5312be6fea7b44fc83d757a6ddc0f1ff" Oct 30 08:07:44.722835 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:07:44.722796 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"947900cbb5ceaac48c9cf3052160be0a5312be6fea7b44fc83d757a6ddc0f1ff\": container with ID starting with 947900cbb5ceaac48c9cf3052160be0a5312be6fea7b44fc83d757a6ddc0f1ff not found: ID does not exist" containerID="947900cbb5ceaac48c9cf3052160be0a5312be6fea7b44fc83d757a6ddc0f1ff" Oct 30 08:07:44.722943 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.722847 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"947900cbb5ceaac48c9cf3052160be0a5312be6fea7b44fc83d757a6ddc0f1ff"} err="failed to get container status \"947900cbb5ceaac48c9cf3052160be0a5312be6fea7b44fc83d757a6ddc0f1ff\": rpc error: code = NotFound desc = could not find container \"947900cbb5ceaac48c9cf3052160be0a5312be6fea7b44fc83d757a6ddc0f1ff\": container with ID starting with 947900cbb5ceaac48c9cf3052160be0a5312be6fea7b44fc83d757a6ddc0f1ff not found: ID does not exist" Oct 30 08:07:44.722943 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.722874 4951 scope.go:122] "RemoveContainer" containerID="5c6a8eec409b079ca0e33fab991eb8bbaae4ac44bce0f48415332c5b33d0dbc2" Oct 30 08:07:44.723149 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:07:44.723127 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c6a8eec409b079ca0e33fab991eb8bbaae4ac44bce0f48415332c5b33d0dbc2\": container with ID starting with 5c6a8eec409b079ca0e33fab991eb8bbaae4ac44bce0f48415332c5b33d0dbc2 not found: ID does not exist" containerID="5c6a8eec409b079ca0e33fab991eb8bbaae4ac44bce0f48415332c5b33d0dbc2" Oct 30 08:07:44.723209 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.723157 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c6a8eec409b079ca0e33fab991eb8bbaae4ac44bce0f48415332c5b33d0dbc2"} err="failed to get container status \"5c6a8eec409b079ca0e33fab991eb8bbaae4ac44bce0f48415332c5b33d0dbc2\": rpc error: code = NotFound desc = could not find container \"5c6a8eec409b079ca0e33fab991eb8bbaae4ac44bce0f48415332c5b33d0dbc2\": container with ID starting with 5c6a8eec409b079ca0e33fab991eb8bbaae4ac44bce0f48415332c5b33d0dbc2 not found: ID does not exist" Oct 30 08:07:44.755633 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.755598 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5"] Oct 30 08:07:44.767671 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:44.767645 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-sp5z5"] Oct 30 08:07:45.773321 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:45.773241 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3" path="/var/lib/kubelet/pods/9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3/volumes" Oct 30 08:07:46.500542 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:46.500508 4951 patch_prober.go:29] interesting pod/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 08:07:46.500731 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:46.500561 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" podUID="b08b669b-c0a7-462c-9b63-aabbb0254dc1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 08:07:47.698987 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.698910 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 08:07:47.722848 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.722802 4951 generic.go:356] "Generic (PLEG): container finished" podID="b08b669b-c0a7-462c-9b63-aabbb0254dc1" containerID="4d0bd6310830aebfaa316fa880e9568454ffeb3bc27c3f2b2a41f603cc76ad57" exitCode=0 Oct 30 08:07:47.723066 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.722856 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" Oct 30 08:07:47.723066 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.722884 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" event={"ID":"b08b669b-c0a7-462c-9b63-aabbb0254dc1","Type":"ContainerDied","Data":"4d0bd6310830aebfaa316fa880e9568454ffeb3bc27c3f2b2a41f603cc76ad57"} Oct 30 08:07:47.723066 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.722951 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc" event={"ID":"b08b669b-c0a7-462c-9b63-aabbb0254dc1","Type":"ContainerDied","Data":"25d313a1081d63874cb163f1f3167e7554db37f47a98dd5e0b0bd4336213f08b"} Oct 30 08:07:47.723066 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.722987 4951 scope.go:122] "RemoveContainer" containerID="4d0bd6310830aebfaa316fa880e9568454ffeb3bc27c3f2b2a41f603cc76ad57" Oct 30 08:07:47.734301 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.734274 4951 scope.go:122] "RemoveContainer" containerID="dfe77b3f6c8dbc03e45aaa35620dc0846e8baba105df5275450257231527b35d" Oct 30 08:07:47.745024 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.744999 4951 scope.go:122] "RemoveContainer" containerID="4d0bd6310830aebfaa316fa880e9568454ffeb3bc27c3f2b2a41f603cc76ad57" Oct 30 08:07:47.745503 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:07:47.745468 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d0bd6310830aebfaa316fa880e9568454ffeb3bc27c3f2b2a41f603cc76ad57\": container with ID starting with 4d0bd6310830aebfaa316fa880e9568454ffeb3bc27c3f2b2a41f603cc76ad57 not found: ID does not exist" containerID="4d0bd6310830aebfaa316fa880e9568454ffeb3bc27c3f2b2a41f603cc76ad57" Oct 30 08:07:47.745667 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.745516 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d0bd6310830aebfaa316fa880e9568454ffeb3bc27c3f2b2a41f603cc76ad57"} err="failed to get container status \"4d0bd6310830aebfaa316fa880e9568454ffeb3bc27c3f2b2a41f603cc76ad57\": rpc error: code = NotFound desc = could not find container \"4d0bd6310830aebfaa316fa880e9568454ffeb3bc27c3f2b2a41f603cc76ad57\": container with ID starting with 4d0bd6310830aebfaa316fa880e9568454ffeb3bc27c3f2b2a41f603cc76ad57 not found: ID does not exist" Oct 30 08:07:47.745667 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.745544 4951 scope.go:122] "RemoveContainer" containerID="dfe77b3f6c8dbc03e45aaa35620dc0846e8baba105df5275450257231527b35d" Oct 30 08:07:47.746452 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:07:47.746420 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfe77b3f6c8dbc03e45aaa35620dc0846e8baba105df5275450257231527b35d\": container with ID starting with dfe77b3f6c8dbc03e45aaa35620dc0846e8baba105df5275450257231527b35d not found: ID does not exist" containerID="dfe77b3f6c8dbc03e45aaa35620dc0846e8baba105df5275450257231527b35d" Oct 30 08:07:47.746633 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.746460 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfe77b3f6c8dbc03e45aaa35620dc0846e8baba105df5275450257231527b35d"} err="failed to get container status \"dfe77b3f6c8dbc03e45aaa35620dc0846e8baba105df5275450257231527b35d\": rpc error: code = NotFound desc = could not find container \"dfe77b3f6c8dbc03e45aaa35620dc0846e8baba105df5275450257231527b35d\": container with ID starting with dfe77b3f6c8dbc03e45aaa35620dc0846e8baba105df5275450257231527b35d not found: ID does not exist" Oct 30 08:07:47.791310 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.791270 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istio-token\" (UniqueName: \"kubernetes.io/projected/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istio-token\") pod \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " Oct 30 08:07:47.791487 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.791467 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istiod-ca-cert\") pod \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " Oct 30 08:07:47.791615 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.791536 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-workload-socket\") pod \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " Oct 30 08:07:47.791689 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.791640 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istio-envoy\") pod \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " Oct 30 08:07:47.791749 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.791694 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-workload-certs\") pod \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " Oct 30 08:07:47.791808 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.791782 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-credential-socket\") pod \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " Oct 30 08:07:47.791882 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.791854 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istio-data\") pod \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " Oct 30 08:07:47.792031 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.792010 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b08b669b-c0a7-462c-9b63-aabbb0254dc1-kube-api-access-n9qhp\" (UniqueName: \"kubernetes.io/projected/b08b669b-c0a7-462c-9b63-aabbb0254dc1-kube-api-access-n9qhp\") pod \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " Oct 30 08:07:47.792113 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.792066 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istio-podinfo\") pod \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\" (UID: \"b08b669b-c0a7-462c-9b63-aabbb0254dc1\") " Oct 30 08:07:47.792290 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.792251 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-workload-certs" pod "b08b669b-c0a7-462c-9b63-aabbb0254dc1" (UID: "b08b669b-c0a7-462c-9b63-aabbb0254dc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:07:47.792475 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.792426 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-credential-socket" pod "b08b669b-c0a7-462c-9b63-aabbb0254dc1" (UID: "b08b669b-c0a7-462c-9b63-aabbb0254dc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:07:47.792884 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.792652 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istio-data" pod "b08b669b-c0a7-462c-9b63-aabbb0254dc1" (UID: "b08b669b-c0a7-462c-9b63-aabbb0254dc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:07:47.792884 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.792728 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:07:47.792884 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.792742 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:07:47.792884 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.792751 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:07:47.793307 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.793273 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istiod-ca-cert" pod "b08b669b-c0a7-462c-9b63-aabbb0254dc1" (UID: "b08b669b-c0a7-462c-9b63-aabbb0254dc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:07:47.793454 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.793336 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-workload-socket" pod "b08b669b-c0a7-462c-9b63-aabbb0254dc1" (UID: "b08b669b-c0a7-462c-9b63-aabbb0254dc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:07:47.794254 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.794228 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istio-token" pod "b08b669b-c0a7-462c-9b63-aabbb0254dc1" (UID: "b08b669b-c0a7-462c-9b63-aabbb0254dc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:07:47.794434 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.794408 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istio-envoy" pod "b08b669b-c0a7-462c-9b63-aabbb0254dc1" (UID: "b08b669b-c0a7-462c-9b63-aabbb0254dc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:07:47.795004 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.794977 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istio-podinfo" pod "b08b669b-c0a7-462c-9b63-aabbb0254dc1" (UID: "b08b669b-c0a7-462c-9b63-aabbb0254dc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:07:47.795086 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.795023 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b08b669b-c0a7-462c-9b63-aabbb0254dc1-kube-api-access-n9qhp" pod "b08b669b-c0a7-462c-9b63-aabbb0254dc1" (UID: "b08b669b-c0a7-462c-9b63-aabbb0254dc1"). InnerVolumeSpecName "kube-api-access-n9qhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:07:47.893826 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.893778 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:07:47.893826 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.893822 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:07:47.893826 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.893837 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:07:47.894137 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.893850 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:07:47.894137 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.893865 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9qhp\" (UniqueName: \"kubernetes.io/projected/b08b669b-c0a7-462c-9b63-aabbb0254dc1-kube-api-access-n9qhp\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:07:47.894137 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:47.893892 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b08b669b-c0a7-462c-9b63-aabbb0254dc1-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:07:48.071281 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:48.071237 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc"] Oct 30 08:07:48.084439 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:48.084397 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-bb2e8f-openshift-default-557964b997-hcrwc"] Oct 30 08:07:49.772675 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:49.772642 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b08b669b-c0a7-462c-9b63-aabbb0254dc1" path="/var/lib/kubelet/pods/b08b669b-c0a7-462c-9b63-aabbb0254dc1/volumes" Oct 30 08:07:53.028817 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:07:53.028775 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 08:07:54.014707 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:54.014666 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1e0880-6946fccf5f-2rcnf" podUID="64ba04d1-bbfe-4ed3-b0b2-55d554a025fb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:07:54.014928 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:54.014684 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-1e0880-79484bb6bd-5vfjq" podUID="33977737-ec70-47e5-a689-4c471c77c42e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:07:54.761000 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:54.760971 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1e0880-79484bb6bd-5vfjq" Oct 30 08:07:54.770967 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:54.770929 4951 generic.go:356] "Generic (PLEG): container finished" podID="64ba04d1-bbfe-4ed3-b0b2-55d554a025fb" containerID="c7c773054675033d5f0b1cbd172417a233d38b94b95b6560c349a423282a997b" exitCode=0 Oct 30 08:07:54.771119 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:54.771027 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1e0880-6946fccf5f-2rcnf" event={"ID":"64ba04d1-bbfe-4ed3-b0b2-55d554a025fb","Type":"ContainerDied","Data":"c7c773054675033d5f0b1cbd172417a233d38b94b95b6560c349a423282a997b"} Oct 30 08:07:54.774397 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:54.774370 4951 generic.go:356] "Generic (PLEG): container finished" podID="33977737-ec70-47e5-a689-4c471c77c42e" containerID="e0cd5e62032638d8f0203a10f7dccac3d09d50a29ee671b3f5abc055bab6b1d9" exitCode=0 Oct 30 08:07:54.774507 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:54.774433 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1e0880-79484bb6bd-5vfjq" Oct 30 08:07:54.774507 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:54.774455 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1e0880-79484bb6bd-5vfjq" event={"ID":"33977737-ec70-47e5-a689-4c471c77c42e","Type":"ContainerDied","Data":"e0cd5e62032638d8f0203a10f7dccac3d09d50a29ee671b3f5abc055bab6b1d9"} Oct 30 08:07:54.774651 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:54.774524 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1e0880-79484bb6bd-5vfjq" event={"ID":"33977737-ec70-47e5-a689-4c471c77c42e","Type":"ContainerDied","Data":"74ffc1b362708be106a837a5e5a89be0e21d575a0d5510c1c611c5806df9a490"} Oct 30 08:07:54.774651 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:54.774560 4951 scope.go:122] "RemoveContainer" containerID="e0cd5e62032638d8f0203a10f7dccac3d09d50a29ee671b3f5abc055bab6b1d9" Oct 30 08:07:54.792009 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:54.791988 4951 scope.go:122] "RemoveContainer" containerID="e0cd5e62032638d8f0203a10f7dccac3d09d50a29ee671b3f5abc055bab6b1d9" Oct 30 08:07:54.792265 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:07:54.792243 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0cd5e62032638d8f0203a10f7dccac3d09d50a29ee671b3f5abc055bab6b1d9\": container with ID starting with e0cd5e62032638d8f0203a10f7dccac3d09d50a29ee671b3f5abc055bab6b1d9 not found: ID does not exist" containerID="e0cd5e62032638d8f0203a10f7dccac3d09d50a29ee671b3f5abc055bab6b1d9" Oct 30 08:07:54.792322 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:54.792275 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0cd5e62032638d8f0203a10f7dccac3d09d50a29ee671b3f5abc055bab6b1d9"} err="failed to get container status \"e0cd5e62032638d8f0203a10f7dccac3d09d50a29ee671b3f5abc055bab6b1d9\": rpc error: code = NotFound desc = could not find container \"e0cd5e62032638d8f0203a10f7dccac3d09d50a29ee671b3f5abc055bab6b1d9\": container with ID starting with e0cd5e62032638d8f0203a10f7dccac3d09d50a29ee671b3f5abc055bab6b1d9 not found: ID does not exist" Oct 30 08:07:54.794973 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:54.794953 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1e0880-6946fccf5f-2rcnf" Oct 30 08:07:54.889281 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:54.889248 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33977737-ec70-47e5-a689-4c471c77c42e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33977737-ec70-47e5-a689-4c471c77c42e-plugins-config-volume\") pod \"33977737-ec70-47e5-a689-4c471c77c42e\" (UID: \"33977737-ec70-47e5-a689-4c471c77c42e\") " Oct 30 08:07:54.889472 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:54.889372 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64ba04d1-bbfe-4ed3-b0b2-55d554a025fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ba04d1-bbfe-4ed3-b0b2-55d554a025fb-plugins-config-volume\") pod \"64ba04d1-bbfe-4ed3-b0b2-55d554a025fb\" (UID: \"64ba04d1-bbfe-4ed3-b0b2-55d554a025fb\") " Oct 30 08:07:54.889472 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:54.889400 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ba04d1-bbfe-4ed3-b0b2-55d554a025fb-kube-api-access-26swz\" (UniqueName: \"kubernetes.io/projected/64ba04d1-bbfe-4ed3-b0b2-55d554a025fb-kube-api-access-26swz\") pod \"64ba04d1-bbfe-4ed3-b0b2-55d554a025fb\" (UID: \"64ba04d1-bbfe-4ed3-b0b2-55d554a025fb\") " Oct 30 08:07:54.889472 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:54.889458 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33977737-ec70-47e5-a689-4c471c77c42e-kube-api-access-95jlm\" (UniqueName: \"kubernetes.io/projected/33977737-ec70-47e5-a689-4c471c77c42e-kube-api-access-95jlm\") pod \"33977737-ec70-47e5-a689-4c471c77c42e\" (UID: \"33977737-ec70-47e5-a689-4c471c77c42e\") " Oct 30 08:07:54.889817 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:54.889786 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33977737-ec70-47e5-a689-4c471c77c42e-plugins-config-volume" pod "33977737-ec70-47e5-a689-4c471c77c42e" (UID: "33977737-ec70-47e5-a689-4c471c77c42e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:07:54.889908 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:54.889844 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ba04d1-bbfe-4ed3-b0b2-55d554a025fb-plugins-config-volume" pod "64ba04d1-bbfe-4ed3-b0b2-55d554a025fb" (UID: "64ba04d1-bbfe-4ed3-b0b2-55d554a025fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:07:54.890114 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:54.890086 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ba04d1-bbfe-4ed3-b0b2-55d554a025fb-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:07:54.890252 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:54.890117 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33977737-ec70-47e5-a689-4c471c77c42e-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:07:54.891765 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:54.891736 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33977737-ec70-47e5-a689-4c471c77c42e-kube-api-access-95jlm" pod "33977737-ec70-47e5-a689-4c471c77c42e" (UID: "33977737-ec70-47e5-a689-4c471c77c42e"). InnerVolumeSpecName "kube-api-access-95jlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:07:54.891876 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:54.891750 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ba04d1-bbfe-4ed3-b0b2-55d554a025fb-kube-api-access-26swz" pod "64ba04d1-bbfe-4ed3-b0b2-55d554a025fb" (UID: "64ba04d1-bbfe-4ed3-b0b2-55d554a025fb"). InnerVolumeSpecName "kube-api-access-26swz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:07:54.991259 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:54.991175 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26swz\" (UniqueName: \"kubernetes.io/projected/64ba04d1-bbfe-4ed3-b0b2-55d554a025fb-kube-api-access-26swz\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:07:54.991259 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:54.991208 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95jlm\" (UniqueName: \"kubernetes.io/projected/33977737-ec70-47e5-a689-4c471c77c42e-kube-api-access-95jlm\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:07:55.125017 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:55.124979 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1e0880-79484bb6bd-5vfjq"] Oct 30 08:07:55.131236 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:55.131207 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1e0880-79484bb6bd-5vfjq"] Oct 30 08:07:55.775646 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:55.775612 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33977737-ec70-47e5-a689-4c471c77c42e" path="/var/lib/kubelet/pods/33977737-ec70-47e5-a689-4c471c77c42e/volumes" Oct 30 08:07:55.785563 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:55.785547 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1e0880-6946fccf5f-2rcnf" Oct 30 08:07:55.785877 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:55.785829 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1e0880-6946fccf5f-2rcnf" event={"ID":"64ba04d1-bbfe-4ed3-b0b2-55d554a025fb","Type":"ContainerDied","Data":"9b626ee5fdbd346ba4d3b2520dd3234309cd4c0a8f15ec86ec71a210055b673b"} Oct 30 08:07:55.786001 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:55.785897 4951 scope.go:122] "RemoveContainer" containerID="c7c773054675033d5f0b1cbd172417a233d38b94b95b6560c349a423282a997b" Oct 30 08:07:55.841117 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:55.841077 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1e0880-6946fccf5f-2rcnf"] Oct 30 08:07:55.845408 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:55.845379 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1e0880-6946fccf5f-2rcnf"] Oct 30 08:07:57.771861 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:07:57.771829 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64ba04d1-bbfe-4ed3-b0b2-55d554a025fb" path="/var/lib/kubelet/pods/64ba04d1-bbfe-4ed3-b0b2-55d554a025fb/volumes" Oct 30 08:08:04.975404 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:04.972953 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf"] Oct 30 08:08:04.986921 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:04.986884 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf" podUID="2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985" containerName="maas-api" containerID="cri-o://26de1a58e5970b358caa8c468faa8d53799278e1664e49343d996487fbd5a423" gracePeriod=30 Oct 30 08:08:05.314475 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:05.314443 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf" Oct 30 08:08:05.351714 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:05.351679 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985-kube-api-access-xkzvw\" (UniqueName: \"kubernetes.io/projected/2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985-kube-api-access-xkzvw\") pod \"2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985\" (UID: \"2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985\") " Oct 30 08:08:05.351942 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:05.351755 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985-metrics-certs\") pod \"2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985\" (UID: \"2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985\") " Oct 30 08:08:05.351942 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:05.351819 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985-openshift-service-ca\") pod \"2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985\" (UID: \"2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985\") " Oct 30 08:08:05.351942 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:05.351926 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985-maas-api-tls\") pod \"2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985\" (UID: \"2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985\") " Oct 30 08:08:05.352288 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:05.352249 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985-openshift-service-ca" pod "2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985" (UID: "2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:08:05.352455 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:05.352430 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:05.354044 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:05.354013 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985-kube-api-access-xkzvw" pod "2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985" (UID: "2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985"). InnerVolumeSpecName "kube-api-access-xkzvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:05.354435 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:05.354409 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985-metrics-certs" pod "2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985" (UID: "2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:08:05.354530 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:05.354438 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985-maas-api-tls" pod "2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985" (UID: "2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:08:05.453771 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:05.453736 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:05.454028 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:05.454010 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkzvw\" (UniqueName: \"kubernetes.io/projected/2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985-kube-api-access-xkzvw\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:05.454148 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:05.454135 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:05.855835 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:05.855796 4951 generic.go:356] "Generic (PLEG): container finished" podID="2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985" containerID="26de1a58e5970b358caa8c468faa8d53799278e1664e49343d996487fbd5a423" exitCode=0 Oct 30 08:08:05.856023 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:05.855848 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf" Oct 30 08:08:05.856023 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:05.855885 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf" event={"ID":"2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985","Type":"ContainerDied","Data":"26de1a58e5970b358caa8c468faa8d53799278e1664e49343d996487fbd5a423"} Oct 30 08:08:05.856023 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:05.855960 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf" event={"ID":"2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985","Type":"ContainerDied","Data":"3a4aff63fd53d7629ced06fea630689cb6dfd357ce048eb47b7a00e6d356b891"} Oct 30 08:08:05.856023 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:05.855991 4951 scope.go:122] "RemoveContainer" containerID="26de1a58e5970b358caa8c468faa8d53799278e1664e49343d996487fbd5a423" Oct 30 08:08:05.868408 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:05.868385 4951 scope.go:122] "RemoveContainer" containerID="26de1a58e5970b358caa8c468faa8d53799278e1664e49343d996487fbd5a423" Oct 30 08:08:05.868823 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:08:05.868787 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26de1a58e5970b358caa8c468faa8d53799278e1664e49343d996487fbd5a423\": container with ID starting with 26de1a58e5970b358caa8c468faa8d53799278e1664e49343d996487fbd5a423 not found: ID does not exist" containerID="26de1a58e5970b358caa8c468faa8d53799278e1664e49343d996487fbd5a423" Oct 30 08:08:05.868983 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:05.868834 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26de1a58e5970b358caa8c468faa8d53799278e1664e49343d996487fbd5a423"} err="failed to get container status \"26de1a58e5970b358caa8c468faa8d53799278e1664e49343d996487fbd5a423\": rpc error: code = NotFound desc = could not find container \"26de1a58e5970b358caa8c468faa8d53799278e1664e49343d996487fbd5a423\": container with ID starting with 26de1a58e5970b358caa8c468faa8d53799278e1664e49343d996487fbd5a423 not found: ID does not exist" Oct 30 08:08:05.912187 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:05.912146 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf"] Oct 30 08:08:05.923553 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:05.923523 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b096ec-65bbfdc9c-77vqf"] Oct 30 08:08:07.770939 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:07.770908 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985" path="/var/lib/kubelet/pods/2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985/volumes" Oct 30 08:08:14.330690 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:14.330654 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8"] Oct 30 08:08:14.332432 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:14.330983 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" podUID="3ad4d41c-7af3-453d-9005-d34264ed3acd" containerName="istio-proxy" containerID="cri-o://a44624c74aedb78858324a1d5e5f3bdd943559c48d8d2a150faf5ea9ceb8822a" gracePeriod=30 Oct 30 08:08:19.600300 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.600228 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 08:08:19.744157 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.744121 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-workload-socket\") pod \"3ad4d41c-7af3-453d-9005-d34264ed3acd\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " Oct 30 08:08:19.744362 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.744197 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ad4d41c-7af3-453d-9005-d34264ed3acd-kube-api-access-nxpzg\" (UniqueName: \"kubernetes.io/projected/3ad4d41c-7af3-453d-9005-d34264ed3acd-kube-api-access-nxpzg\") pod \"3ad4d41c-7af3-453d-9005-d34264ed3acd\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " Oct 30 08:08:19.744362 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.744249 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-credential-socket\") pod \"3ad4d41c-7af3-453d-9005-d34264ed3acd\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " Oct 30 08:08:19.744362 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.744285 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ad4d41c-7af3-453d-9005-d34264ed3acd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad4d41c-7af3-453d-9005-d34264ed3acd-istio-podinfo\") pod \"3ad4d41c-7af3-453d-9005-d34264ed3acd\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " Oct 30 08:08:19.744362 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.744353 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-workload-certs\") pod \"3ad4d41c-7af3-453d-9005-d34264ed3acd\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " Oct 30 08:08:19.744657 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.744430 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ad4d41c-7af3-453d-9005-d34264ed3acd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad4d41c-7af3-453d-9005-d34264ed3acd-istiod-ca-cert\") pod \"3ad4d41c-7af3-453d-9005-d34264ed3acd\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " Oct 30 08:08:19.744657 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.744488 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-istio-envoy\") pod \"3ad4d41c-7af3-453d-9005-d34264ed3acd\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " Oct 30 08:08:19.744657 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.744544 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-istio-data\") pod \"3ad4d41c-7af3-453d-9005-d34264ed3acd\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " Oct 30 08:08:19.744657 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.744425 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-workload-socket" pod "3ad4d41c-7af3-453d-9005-d34264ed3acd" (UID: "3ad4d41c-7af3-453d-9005-d34264ed3acd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:08:19.744657 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.744584 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-credential-socket" pod "3ad4d41c-7af3-453d-9005-d34264ed3acd" (UID: "3ad4d41c-7af3-453d-9005-d34264ed3acd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:08:19.744657 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.744644 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ad4d41c-7af3-453d-9005-d34264ed3acd-istio-token\" (UniqueName: \"kubernetes.io/projected/3ad4d41c-7af3-453d-9005-d34264ed3acd-istio-token\") pod \"3ad4d41c-7af3-453d-9005-d34264ed3acd\" (UID: \"3ad4d41c-7af3-453d-9005-d34264ed3acd\") " Oct 30 08:08:19.744959 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.744670 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-workload-certs" pod "3ad4d41c-7af3-453d-9005-d34264ed3acd" (UID: "3ad4d41c-7af3-453d-9005-d34264ed3acd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:08:19.744959 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.744794 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ad4d41c-7af3-453d-9005-d34264ed3acd-istiod-ca-cert" pod "3ad4d41c-7af3-453d-9005-d34264ed3acd" (UID: "3ad4d41c-7af3-453d-9005-d34264ed3acd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:08:19.745072 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.745054 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-istio-data" pod "3ad4d41c-7af3-453d-9005-d34264ed3acd" (UID: "3ad4d41c-7af3-453d-9005-d34264ed3acd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:08:19.745256 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.745229 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:19.745326 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.745257 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:19.745326 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.745272 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:19.745326 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.745286 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad4d41c-7af3-453d-9005-d34264ed3acd-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:19.745326 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.745298 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:19.746857 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.746829 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-istio-envoy" pod "3ad4d41c-7af3-453d-9005-d34264ed3acd" (UID: "3ad4d41c-7af3-453d-9005-d34264ed3acd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:08:19.747178 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.747155 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ad4d41c-7af3-453d-9005-d34264ed3acd-istio-podinfo" pod "3ad4d41c-7af3-453d-9005-d34264ed3acd" (UID: "3ad4d41c-7af3-453d-9005-d34264ed3acd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:08:19.747262 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.747200 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ad4d41c-7af3-453d-9005-d34264ed3acd-kube-api-access-nxpzg" pod "3ad4d41c-7af3-453d-9005-d34264ed3acd" (UID: "3ad4d41c-7af3-453d-9005-d34264ed3acd"). InnerVolumeSpecName "kube-api-access-nxpzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:19.747262 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.747225 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ad4d41c-7af3-453d-9005-d34264ed3acd-istio-token" pod "3ad4d41c-7af3-453d-9005-d34264ed3acd" (UID: "3ad4d41c-7af3-453d-9005-d34264ed3acd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:19.846217 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.846185 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad4d41c-7af3-453d-9005-d34264ed3acd-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:19.846217 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.846217 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad4d41c-7af3-453d-9005-d34264ed3acd-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:19.846217 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.846230 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxpzg\" (UniqueName: \"kubernetes.io/projected/3ad4d41c-7af3-453d-9005-d34264ed3acd-kube-api-access-nxpzg\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:19.846473 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.846243 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad4d41c-7af3-453d-9005-d34264ed3acd-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:19.947748 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.947663 4951 generic.go:356] "Generic (PLEG): container finished" podID="3ad4d41c-7af3-453d-9005-d34264ed3acd" containerID="a44624c74aedb78858324a1d5e5f3bdd943559c48d8d2a150faf5ea9ceb8822a" exitCode=0 Oct 30 08:08:19.947748 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.947715 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" Oct 30 08:08:19.947972 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.947740 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" event={"ID":"3ad4d41c-7af3-453d-9005-d34264ed3acd","Type":"ContainerDied","Data":"a44624c74aedb78858324a1d5e5f3bdd943559c48d8d2a150faf5ea9ceb8822a"} Oct 30 08:08:19.947972 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.947799 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8" event={"ID":"3ad4d41c-7af3-453d-9005-d34264ed3acd","Type":"ContainerDied","Data":"86a9584aa9c2aca2c746bdc80326d7290b4aab6c5a224c186638a03c383249de"} Oct 30 08:08:19.947972 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.947820 4951 scope.go:122] "RemoveContainer" containerID="a44624c74aedb78858324a1d5e5f3bdd943559c48d8d2a150faf5ea9ceb8822a" Oct 30 08:08:19.957477 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.957444 4951 scope.go:122] "RemoveContainer" containerID="818324153ca59fed54d04cb66f35f25f4dcb53b96e013b9b2fa0690353ed7c60" Oct 30 08:08:19.969745 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.969712 4951 scope.go:122] "RemoveContainer" containerID="a44624c74aedb78858324a1d5e5f3bdd943559c48d8d2a150faf5ea9ceb8822a" Oct 30 08:08:19.970274 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:08:19.970244 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a44624c74aedb78858324a1d5e5f3bdd943559c48d8d2a150faf5ea9ceb8822a\": container with ID starting with a44624c74aedb78858324a1d5e5f3bdd943559c48d8d2a150faf5ea9ceb8822a not found: ID does not exist" containerID="a44624c74aedb78858324a1d5e5f3bdd943559c48d8d2a150faf5ea9ceb8822a" Oct 30 08:08:19.970387 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.970289 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a44624c74aedb78858324a1d5e5f3bdd943559c48d8d2a150faf5ea9ceb8822a"} err="failed to get container status \"a44624c74aedb78858324a1d5e5f3bdd943559c48d8d2a150faf5ea9ceb8822a\": rpc error: code = NotFound desc = could not find container \"a44624c74aedb78858324a1d5e5f3bdd943559c48d8d2a150faf5ea9ceb8822a\": container with ID starting with a44624c74aedb78858324a1d5e5f3bdd943559c48d8d2a150faf5ea9ceb8822a not found: ID does not exist" Oct 30 08:08:19.970387 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.970317 4951 scope.go:122] "RemoveContainer" containerID="818324153ca59fed54d04cb66f35f25f4dcb53b96e013b9b2fa0690353ed7c60" Oct 30 08:08:19.970645 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:08:19.970621 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"818324153ca59fed54d04cb66f35f25f4dcb53b96e013b9b2fa0690353ed7c60\": container with ID starting with 818324153ca59fed54d04cb66f35f25f4dcb53b96e013b9b2fa0690353ed7c60 not found: ID does not exist" containerID="818324153ca59fed54d04cb66f35f25f4dcb53b96e013b9b2fa0690353ed7c60" Oct 30 08:08:19.970697 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:19.970653 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"818324153ca59fed54d04cb66f35f25f4dcb53b96e013b9b2fa0690353ed7c60"} err="failed to get container status \"818324153ca59fed54d04cb66f35f25f4dcb53b96e013b9b2fa0690353ed7c60\": rpc error: code = NotFound desc = could not find container \"818324153ca59fed54d04cb66f35f25f4dcb53b96e013b9b2fa0690353ed7c60\": container with ID starting with 818324153ca59fed54d04cb66f35f25f4dcb53b96e013b9b2fa0690353ed7c60 not found: ID does not exist" Oct 30 08:08:20.003311 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:20.003280 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8"] Oct 30 08:08:20.007211 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:20.007179 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-b096ec-openshift-default-5f47db78cc-kqmr8"] Oct 30 08:08:21.291263 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:21.291227 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb"] Oct 30 08:08:21.291825 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:21.291472 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" podUID="3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b" containerName="istio-proxy" containerID="cri-o://6afb17f54968c384715f464019e870ac9acb55cbcd700372d5fa74d706553fa7" gracePeriod=30 Oct 30 08:08:21.771038 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:21.771000 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ad4d41c-7af3-453d-9005-d34264ed3acd" path="/var/lib/kubelet/pods/3ad4d41c-7af3-453d-9005-d34264ed3acd/volumes" Oct 30 08:08:23.023780 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:08:23.023741 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 08:08:23.117810 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:23.117775 4951 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 08:08:23.118021 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:23.117832 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" podUID="3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 08:08:26.573138 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:26.573069 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 08:08:26.732257 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:26.732222 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istio-envoy\") pod \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " Oct 30 08:08:26.732472 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:26.732268 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istio-token\" (UniqueName: \"kubernetes.io/projected/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istio-token\") pod \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " Oct 30 08:08:26.732472 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:26.732301 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-kube-api-access-k8wc9\" (UniqueName: \"kubernetes.io/projected/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-kube-api-access-k8wc9\") pod \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " Oct 30 08:08:26.732472 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:26.732331 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-credential-socket\") pod \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " Oct 30 08:08:26.732655 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:26.732488 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istiod-ca-cert\") pod \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " Oct 30 08:08:26.732655 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:26.732552 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-workload-socket\") pod \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " Oct 30 08:08:26.732748 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:26.732678 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-workload-certs\") pod \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " Oct 30 08:08:26.732795 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:26.732755 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istio-data\") pod \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " Oct 30 08:08:26.732833 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:26.732797 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istio-podinfo\") pod \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\" (UID: \"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b\") " Oct 30 08:08:26.733620 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:26.733590 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-workload-certs" pod "3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b" (UID: "3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:08:26.733753 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:26.733623 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-credential-socket" pod "3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b" (UID: "3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:08:26.733753 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:26.733654 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istiod-ca-cert" pod "3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b" (UID: "3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:08:26.733753 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:26.733717 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-workload-socket" pod "3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b" (UID: "3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:08:26.733901 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:26.733827 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istio-data" pod "3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b" (UID: "3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:08:26.734878 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:26.734852 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istio-envoy" pod "3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b" (UID: "3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:08:26.735500 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:26.735471 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istio-token" pod "3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b" (UID: "3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:26.735851 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:26.735827 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istio-podinfo" pod "3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b" (UID: "3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:08:26.735958 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:26.735938 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-kube-api-access-k8wc9" pod "3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b" (UID: "3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b"). InnerVolumeSpecName "kube-api-access-k8wc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:26.834145 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:26.834056 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:26.834145 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:26.834086 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:26.834145 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:26.834098 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8wc9\" (UniqueName: \"kubernetes.io/projected/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-kube-api-access-k8wc9\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:26.834145 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:26.834111 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:26.834145 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:26.834123 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:26.834145 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:26.834134 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:26.834145 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:26.834148 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:26.834492 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:26.834160 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:26.834492 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:26.834171 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:26.995396 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:26.995355 4951 generic.go:356] "Generic (PLEG): container finished" podID="3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b" containerID="6afb17f54968c384715f464019e870ac9acb55cbcd700372d5fa74d706553fa7" exitCode=0 Oct 30 08:08:26.995610 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:26.995415 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" Oct 30 08:08:26.995610 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:26.995433 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" event={"ID":"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b","Type":"ContainerDied","Data":"6afb17f54968c384715f464019e870ac9acb55cbcd700372d5fa74d706553fa7"} Oct 30 08:08:26.995610 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:26.995493 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb" event={"ID":"3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b","Type":"ContainerDied","Data":"c3df55d0f8975547f37c10419f043710583a1821726bf132a5f5c210a951f9f4"} Oct 30 08:08:26.995610 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:26.995513 4951 scope.go:122] "RemoveContainer" containerID="6afb17f54968c384715f464019e870ac9acb55cbcd700372d5fa74d706553fa7" Oct 30 08:08:27.011981 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:27.011952 4951 scope.go:122] "RemoveContainer" containerID="5feecbec4c9580ec0597e82af6032e5943d3900344ca4aeae190968dd7b027e6" Oct 30 08:08:27.023074 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:27.023052 4951 scope.go:122] "RemoveContainer" containerID="6afb17f54968c384715f464019e870ac9acb55cbcd700372d5fa74d706553fa7" Oct 30 08:08:27.023439 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:08:27.023409 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6afb17f54968c384715f464019e870ac9acb55cbcd700372d5fa74d706553fa7\": container with ID starting with 6afb17f54968c384715f464019e870ac9acb55cbcd700372d5fa74d706553fa7 not found: ID does not exist" containerID="6afb17f54968c384715f464019e870ac9acb55cbcd700372d5fa74d706553fa7" Oct 30 08:08:27.023544 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:27.023449 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6afb17f54968c384715f464019e870ac9acb55cbcd700372d5fa74d706553fa7"} err="failed to get container status \"6afb17f54968c384715f464019e870ac9acb55cbcd700372d5fa74d706553fa7\": rpc error: code = NotFound desc = could not find container \"6afb17f54968c384715f464019e870ac9acb55cbcd700372d5fa74d706553fa7\": container with ID starting with 6afb17f54968c384715f464019e870ac9acb55cbcd700372d5fa74d706553fa7 not found: ID does not exist" Oct 30 08:08:27.023544 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:27.023475 4951 scope.go:122] "RemoveContainer" containerID="5feecbec4c9580ec0597e82af6032e5943d3900344ca4aeae190968dd7b027e6" Oct 30 08:08:27.023777 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:08:27.023754 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5feecbec4c9580ec0597e82af6032e5943d3900344ca4aeae190968dd7b027e6\": container with ID starting with 5feecbec4c9580ec0597e82af6032e5943d3900344ca4aeae190968dd7b027e6 not found: ID does not exist" containerID="5feecbec4c9580ec0597e82af6032e5943d3900344ca4aeae190968dd7b027e6" Oct 30 08:08:27.023836 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:27.023784 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5feecbec4c9580ec0597e82af6032e5943d3900344ca4aeae190968dd7b027e6"} err="failed to get container status \"5feecbec4c9580ec0597e82af6032e5943d3900344ca4aeae190968dd7b027e6\": rpc error: code = NotFound desc = could not find container \"5feecbec4c9580ec0597e82af6032e5943d3900344ca4aeae190968dd7b027e6\": container with ID starting with 5feecbec4c9580ec0597e82af6032e5943d3900344ca4aeae190968dd7b027e6 not found: ID does not exist" Oct 30 08:08:27.049012 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:27.048987 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb"] Oct 30 08:08:27.053195 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:27.053174 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d89593-openshift-default-777fd554f7-nggsb"] Oct 30 08:08:27.771530 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:27.771493 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b" path="/var/lib/kubelet/pods/3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b/volumes" Oct 30 08:08:28.288811 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:28.288774 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst"] Oct 30 08:08:28.289091 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:28.289062 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst" podUID="2149f8b0-9707-4e35-851e-af05b7842bf3" containerName="maas-api" containerID="cri-o://d68cb0c0eb70fc03f9f95b0287047f21cb273108e7294f9a6709ef486cddb9c1" gracePeriod=30 Oct 30 08:08:28.587631 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:28.587595 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst" Oct 30 08:08:28.762843 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:28.761922 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2149f8b0-9707-4e35-851e-af05b7842bf3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2149f8b0-9707-4e35-851e-af05b7842bf3-metrics-certs\") pod \"2149f8b0-9707-4e35-851e-af05b7842bf3\" (UID: \"2149f8b0-9707-4e35-851e-af05b7842bf3\") " Oct 30 08:08:28.762843 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:28.762006 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2149f8b0-9707-4e35-851e-af05b7842bf3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2149f8b0-9707-4e35-851e-af05b7842bf3-maas-api-tls\") pod \"2149f8b0-9707-4e35-851e-af05b7842bf3\" (UID: \"2149f8b0-9707-4e35-851e-af05b7842bf3\") " Oct 30 08:08:28.762843 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:28.762137 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2149f8b0-9707-4e35-851e-af05b7842bf3-kube-api-access-nfkln\" (UniqueName: \"kubernetes.io/projected/2149f8b0-9707-4e35-851e-af05b7842bf3-kube-api-access-nfkln\") pod \"2149f8b0-9707-4e35-851e-af05b7842bf3\" (UID: \"2149f8b0-9707-4e35-851e-af05b7842bf3\") " Oct 30 08:08:28.762843 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:28.762230 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2149f8b0-9707-4e35-851e-af05b7842bf3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2149f8b0-9707-4e35-851e-af05b7842bf3-openshift-service-ca\") pod \"2149f8b0-9707-4e35-851e-af05b7842bf3\" (UID: \"2149f8b0-9707-4e35-851e-af05b7842bf3\") " Oct 30 08:08:28.763233 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:28.762944 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2149f8b0-9707-4e35-851e-af05b7842bf3-openshift-service-ca" pod "2149f8b0-9707-4e35-851e-af05b7842bf3" (UID: "2149f8b0-9707-4e35-851e-af05b7842bf3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:08:28.765772 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:28.765737 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2149f8b0-9707-4e35-851e-af05b7842bf3-metrics-certs" pod "2149f8b0-9707-4e35-851e-af05b7842bf3" (UID: "2149f8b0-9707-4e35-851e-af05b7842bf3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:08:28.775759 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:28.775696 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2149f8b0-9707-4e35-851e-af05b7842bf3-kube-api-access-nfkln" pod "2149f8b0-9707-4e35-851e-af05b7842bf3" (UID: "2149f8b0-9707-4e35-851e-af05b7842bf3"). InnerVolumeSpecName "kube-api-access-nfkln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:28.776857 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:28.776823 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2149f8b0-9707-4e35-851e-af05b7842bf3-maas-api-tls" pod "2149f8b0-9707-4e35-851e-af05b7842bf3" (UID: "2149f8b0-9707-4e35-851e-af05b7842bf3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:08:28.864213 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:28.864128 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2149f8b0-9707-4e35-851e-af05b7842bf3-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:28.864213 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:28.864157 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2149f8b0-9707-4e35-851e-af05b7842bf3-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:28.864213 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:28.864170 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfkln\" (UniqueName: \"kubernetes.io/projected/2149f8b0-9707-4e35-851e-af05b7842bf3-kube-api-access-nfkln\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:28.864213 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:28.864183 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2149f8b0-9707-4e35-851e-af05b7842bf3-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:29.015513 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:29.015471 4951 generic.go:356] "Generic (PLEG): container finished" podID="2149f8b0-9707-4e35-851e-af05b7842bf3" containerID="d68cb0c0eb70fc03f9f95b0287047f21cb273108e7294f9a6709ef486cddb9c1" exitCode=0 Oct 30 08:08:29.015763 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:29.015523 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst" Oct 30 08:08:29.015763 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:29.015554 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst" event={"ID":"2149f8b0-9707-4e35-851e-af05b7842bf3","Type":"ContainerDied","Data":"d68cb0c0eb70fc03f9f95b0287047f21cb273108e7294f9a6709ef486cddb9c1"} Oct 30 08:08:29.015763 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:29.015653 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst" event={"ID":"2149f8b0-9707-4e35-851e-af05b7842bf3","Type":"ContainerDied","Data":"49650bfff22b4994fbf9d477cd4172a00d19f3fb331fe8ed522fdf4d18861b9a"} Oct 30 08:08:29.015763 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:29.015684 4951 scope.go:122] "RemoveContainer" containerID="d68cb0c0eb70fc03f9f95b0287047f21cb273108e7294f9a6709ef486cddb9c1" Oct 30 08:08:29.026745 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:29.026708 4951 scope.go:122] "RemoveContainer" containerID="d68cb0c0eb70fc03f9f95b0287047f21cb273108e7294f9a6709ef486cddb9c1" Oct 30 08:08:29.027036 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:08:29.027013 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d68cb0c0eb70fc03f9f95b0287047f21cb273108e7294f9a6709ef486cddb9c1\": container with ID starting with d68cb0c0eb70fc03f9f95b0287047f21cb273108e7294f9a6709ef486cddb9c1 not found: ID does not exist" containerID="d68cb0c0eb70fc03f9f95b0287047f21cb273108e7294f9a6709ef486cddb9c1" Oct 30 08:08:29.027132 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:29.027052 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d68cb0c0eb70fc03f9f95b0287047f21cb273108e7294f9a6709ef486cddb9c1"} err="failed to get container status \"d68cb0c0eb70fc03f9f95b0287047f21cb273108e7294f9a6709ef486cddb9c1\": rpc error: code = NotFound desc = could not find container \"d68cb0c0eb70fc03f9f95b0287047f21cb273108e7294f9a6709ef486cddb9c1\": container with ID starting with d68cb0c0eb70fc03f9f95b0287047f21cb273108e7294f9a6709ef486cddb9c1 not found: ID does not exist" Oct 30 08:08:29.073016 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:29.072986 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst"] Oct 30 08:08:29.085466 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:29.085440 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0f76b0-5ddf4499c4-vwmst"] Oct 30 08:08:29.772528 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:29.772480 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2149f8b0-9707-4e35-851e-af05b7842bf3" path="/var/lib/kubelet/pods/2149f8b0-9707-4e35-851e-af05b7842bf3/volumes" Oct 30 08:08:32.332055 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:32.332010 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-054ef1-5dfd5878db-td8f2"] Oct 30 08:08:32.369424 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:32.369388 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-054ef1-5b764f8cf-ms552"] Oct 30 08:08:35.598536 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:35.598497 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt"] Oct 30 08:08:35.599067 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:35.598735 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt" podUID="147db729-cd3c-40f4-ae95-dd4db5869e60" containerName="maas-api" containerID="cri-o://596c59fe2d3ef541b268b3701ac4ebf1cb14b9d259753c1d345f36cc1ee4ed21" gracePeriod=30 Oct 30 08:08:35.894675 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:35.894647 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt" Oct 30 08:08:35.943604 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:35.942511 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/147db729-cd3c-40f4-ae95-dd4db5869e60-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/147db729-cd3c-40f4-ae95-dd4db5869e60-maas-api-tls\") pod \"147db729-cd3c-40f4-ae95-dd4db5869e60\" (UID: \"147db729-cd3c-40f4-ae95-dd4db5869e60\") " Oct 30 08:08:35.943604 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:35.942598 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/147db729-cd3c-40f4-ae95-dd4db5869e60-kube-api-access-2rzxd\" (UniqueName: \"kubernetes.io/projected/147db729-cd3c-40f4-ae95-dd4db5869e60-kube-api-access-2rzxd\") pod \"147db729-cd3c-40f4-ae95-dd4db5869e60\" (UID: \"147db729-cd3c-40f4-ae95-dd4db5869e60\") " Oct 30 08:08:35.943604 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:35.942662 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/147db729-cd3c-40f4-ae95-dd4db5869e60-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/147db729-cd3c-40f4-ae95-dd4db5869e60-openshift-service-ca\") pod \"147db729-cd3c-40f4-ae95-dd4db5869e60\" (UID: \"147db729-cd3c-40f4-ae95-dd4db5869e60\") " Oct 30 08:08:35.943604 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:35.942694 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/147db729-cd3c-40f4-ae95-dd4db5869e60-metrics-certs\" (UniqueName: \"kubernetes.io/secret/147db729-cd3c-40f4-ae95-dd4db5869e60-metrics-certs\") pod \"147db729-cd3c-40f4-ae95-dd4db5869e60\" (UID: \"147db729-cd3c-40f4-ae95-dd4db5869e60\") " Oct 30 08:08:35.943604 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:35.943023 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/147db729-cd3c-40f4-ae95-dd4db5869e60-openshift-service-ca" pod "147db729-cd3c-40f4-ae95-dd4db5869e60" (UID: "147db729-cd3c-40f4-ae95-dd4db5869e60"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:08:35.943604 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:35.943111 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/147db729-cd3c-40f4-ae95-dd4db5869e60-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:35.945274 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:35.945236 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/147db729-cd3c-40f4-ae95-dd4db5869e60-maas-api-tls" pod "147db729-cd3c-40f4-ae95-dd4db5869e60" (UID: "147db729-cd3c-40f4-ae95-dd4db5869e60"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:08:35.946400 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:35.946363 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147db729-cd3c-40f4-ae95-dd4db5869e60-kube-api-access-2rzxd" pod "147db729-cd3c-40f4-ae95-dd4db5869e60" (UID: "147db729-cd3c-40f4-ae95-dd4db5869e60"). InnerVolumeSpecName "kube-api-access-2rzxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:35.952317 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:35.952280 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/147db729-cd3c-40f4-ae95-dd4db5869e60-metrics-certs" pod "147db729-cd3c-40f4-ae95-dd4db5869e60" (UID: "147db729-cd3c-40f4-ae95-dd4db5869e60"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:08:36.044125 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:36.044080 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/147db729-cd3c-40f4-ae95-dd4db5869e60-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:36.044125 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:36.044119 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rzxd\" (UniqueName: \"kubernetes.io/projected/147db729-cd3c-40f4-ae95-dd4db5869e60-kube-api-access-2rzxd\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:36.044436 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:36.044133 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/147db729-cd3c-40f4-ae95-dd4db5869e60-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:36.091703 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:36.091499 4951 generic.go:356] "Generic (PLEG): container finished" podID="147db729-cd3c-40f4-ae95-dd4db5869e60" containerID="596c59fe2d3ef541b268b3701ac4ebf1cb14b9d259753c1d345f36cc1ee4ed21" exitCode=0 Oct 30 08:08:36.091703 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:36.091588 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt" Oct 30 08:08:36.091703 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:36.091552 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt" event={"ID":"147db729-cd3c-40f4-ae95-dd4db5869e60","Type":"ContainerDied","Data":"596c59fe2d3ef541b268b3701ac4ebf1cb14b9d259753c1d345f36cc1ee4ed21"} Oct 30 08:08:36.091703 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:36.091692 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt" event={"ID":"147db729-cd3c-40f4-ae95-dd4db5869e60","Type":"ContainerDied","Data":"46fe1c49f1ebea1864124205389a22097568288524e29eec1cd1372097d8c4c7"} Oct 30 08:08:36.092084 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:36.091726 4951 scope.go:122] "RemoveContainer" containerID="596c59fe2d3ef541b268b3701ac4ebf1cb14b9d259753c1d345f36cc1ee4ed21" Oct 30 08:08:36.103223 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:36.103203 4951 scope.go:122] "RemoveContainer" containerID="596c59fe2d3ef541b268b3701ac4ebf1cb14b9d259753c1d345f36cc1ee4ed21" Oct 30 08:08:36.103521 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:08:36.103503 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"596c59fe2d3ef541b268b3701ac4ebf1cb14b9d259753c1d345f36cc1ee4ed21\": container with ID starting with 596c59fe2d3ef541b268b3701ac4ebf1cb14b9d259753c1d345f36cc1ee4ed21 not found: ID does not exist" containerID="596c59fe2d3ef541b268b3701ac4ebf1cb14b9d259753c1d345f36cc1ee4ed21" Oct 30 08:08:36.103627 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:36.103531 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"596c59fe2d3ef541b268b3701ac4ebf1cb14b9d259753c1d345f36cc1ee4ed21"} err="failed to get container status \"596c59fe2d3ef541b268b3701ac4ebf1cb14b9d259753c1d345f36cc1ee4ed21\": rpc error: code = NotFound desc = could not find container \"596c59fe2d3ef541b268b3701ac4ebf1cb14b9d259753c1d345f36cc1ee4ed21\": container with ID starting with 596c59fe2d3ef541b268b3701ac4ebf1cb14b9d259753c1d345f36cc1ee4ed21 not found: ID does not exist" Oct 30 08:08:36.148997 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:36.148922 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt"] Oct 30 08:08:36.153285 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:36.153259 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-054ef1-656b698f97-rcqqt"] Oct 30 08:08:37.332635 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:37.332546 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-054ef1-5dfd5878db-td8f2" podUID="8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc" containerName="payload-processing" containerID="cri-o://12800c6ba21910303e830d18aef96448ad7d17b3f8e26849cd8adc165cde5467" gracePeriod=25 Oct 30 08:08:37.370291 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:37.370239 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-054ef1-5b764f8cf-ms552" podUID="070057c0-00c3-49dc-a4d2-868885c1685c" containerName="payload-pre-processing" containerID="cri-o://ab330af512d31254a3ea895c288da180a4869d56a8d3976ef5d0698345ce2ea3" gracePeriod=25 Oct 30 08:08:37.772233 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:37.772202 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="147db729-cd3c-40f4-ae95-dd4db5869e60" path="/var/lib/kubelet/pods/147db729-cd3c-40f4-ae95-dd4db5869e60/volumes" Oct 30 08:08:37.912304 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:37.912269 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg"] Oct 30 08:08:37.912559 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:37.912511 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg" podUID="75cf7bf0-7eb7-4f6c-a5a1-c9436acec579" containerName="maas-api" containerID="cri-o://ffd6e250c5965c0eff17c2781ae999596227151ff676c190a7da11663c7d51fe" gracePeriod=30 Oct 30 08:08:38.114556 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:38.114522 4951 generic.go:356] "Generic (PLEG): container finished" podID="75cf7bf0-7eb7-4f6c-a5a1-c9436acec579" containerID="ffd6e250c5965c0eff17c2781ae999596227151ff676c190a7da11663c7d51fe" exitCode=0 Oct 30 08:08:38.114776 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:38.114603 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg" event={"ID":"75cf7bf0-7eb7-4f6c-a5a1-c9436acec579","Type":"ContainerDied","Data":"ffd6e250c5965c0eff17c2781ae999596227151ff676c190a7da11663c7d51fe"} Oct 30 08:08:38.194979 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:38.194892 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg" Oct 30 08:08:38.269752 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:38.269716 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75cf7bf0-7eb7-4f6c-a5a1-c9436acec579-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75cf7bf0-7eb7-4f6c-a5a1-c9436acec579-maas-api-tls\") pod \"75cf7bf0-7eb7-4f6c-a5a1-c9436acec579\" (UID: \"75cf7bf0-7eb7-4f6c-a5a1-c9436acec579\") " Oct 30 08:08:38.269926 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:38.269847 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75cf7bf0-7eb7-4f6c-a5a1-c9436acec579-kube-api-access-vhlxl\" (UniqueName: \"kubernetes.io/projected/75cf7bf0-7eb7-4f6c-a5a1-c9436acec579-kube-api-access-vhlxl\") pod \"75cf7bf0-7eb7-4f6c-a5a1-c9436acec579\" (UID: \"75cf7bf0-7eb7-4f6c-a5a1-c9436acec579\") " Oct 30 08:08:38.270059 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:38.270037 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75cf7bf0-7eb7-4f6c-a5a1-c9436acec579-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75cf7bf0-7eb7-4f6c-a5a1-c9436acec579-openshift-service-ca\") pod \"75cf7bf0-7eb7-4f6c-a5a1-c9436acec579\" (UID: \"75cf7bf0-7eb7-4f6c-a5a1-c9436acec579\") " Oct 30 08:08:38.270176 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:38.270160 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75cf7bf0-7eb7-4f6c-a5a1-c9436acec579-metrics-certs\" (UniqueName: \"kubernetes.io/secret/75cf7bf0-7eb7-4f6c-a5a1-c9436acec579-metrics-certs\") pod \"75cf7bf0-7eb7-4f6c-a5a1-c9436acec579\" (UID: \"75cf7bf0-7eb7-4f6c-a5a1-c9436acec579\") " Oct 30 08:08:38.270609 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:38.270439 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75cf7bf0-7eb7-4f6c-a5a1-c9436acec579-openshift-service-ca" pod "75cf7bf0-7eb7-4f6c-a5a1-c9436acec579" (UID: "75cf7bf0-7eb7-4f6c-a5a1-c9436acec579"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:08:38.270734 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:38.270686 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75cf7bf0-7eb7-4f6c-a5a1-c9436acec579-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:38.272310 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:38.272273 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75cf7bf0-7eb7-4f6c-a5a1-c9436acec579-metrics-certs" pod "75cf7bf0-7eb7-4f6c-a5a1-c9436acec579" (UID: "75cf7bf0-7eb7-4f6c-a5a1-c9436acec579"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:08:38.272597 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:38.272549 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75cf7bf0-7eb7-4f6c-a5a1-c9436acec579-kube-api-access-vhlxl" pod "75cf7bf0-7eb7-4f6c-a5a1-c9436acec579" (UID: "75cf7bf0-7eb7-4f6c-a5a1-c9436acec579"). InnerVolumeSpecName "kube-api-access-vhlxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:38.272678 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:38.272626 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75cf7bf0-7eb7-4f6c-a5a1-c9436acec579-maas-api-tls" pod "75cf7bf0-7eb7-4f6c-a5a1-c9436acec579" (UID: "75cf7bf0-7eb7-4f6c-a5a1-c9436acec579"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:08:38.371480 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:38.371440 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhlxl\" (UniqueName: \"kubernetes.io/projected/75cf7bf0-7eb7-4f6c-a5a1-c9436acec579-kube-api-access-vhlxl\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:38.371480 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:38.371480 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75cf7bf0-7eb7-4f6c-a5a1-c9436acec579-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:38.371962 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:38.371495 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75cf7bf0-7eb7-4f6c-a5a1-c9436acec579-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:39.122149 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:39.122110 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg" Oct 30 08:08:39.122386 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:39.122146 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg" event={"ID":"75cf7bf0-7eb7-4f6c-a5a1-c9436acec579","Type":"ContainerDied","Data":"637326e4a89863abdcba84f3ade3c20e40d08e1393bab2bf80d3f64777343cbd"} Oct 30 08:08:39.122386 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:39.122217 4951 scope.go:122] "RemoveContainer" containerID="ffd6e250c5965c0eff17c2781ae999596227151ff676c190a7da11663c7d51fe" Oct 30 08:08:39.179513 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:39.179476 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg"] Oct 30 08:08:39.192117 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:39.192085 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2e8b2a-848bc45fbd-q5ljg"] Oct 30 08:08:39.771782 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:39.771748 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75cf7bf0-7eb7-4f6c-a5a1-c9436acec579" path="/var/lib/kubelet/pods/75cf7bf0-7eb7-4f6c-a5a1-c9436acec579/volumes" Oct 30 08:08:40.261980 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:40.261946 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-054ef1-5dfd5878db-td8f2" podUID="8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:08:41.272456 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:41.272412 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-054ef1-5b764f8cf-ms552" podUID="070057c0-00c3-49dc-a4d2-868885c1685c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:08:46.741191 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:46.741156 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5"] Oct 30 08:08:46.741708 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:46.741384 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" podUID="85d10cb3-066c-46fa-a149-209a92a409df" containerName="istio-proxy" containerID="cri-o://d539cd44faf50453ea041b22e793b75fd11a246981b674503412043d37ad6a60" gracePeriod=30 Oct 30 08:08:49.223388 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:49.223351 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2"] Oct 30 08:08:49.252662 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:49.252629 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr"] Oct 30 08:08:50.261195 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:50.261151 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-054ef1-5dfd5878db-td8f2" podUID="8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:08:51.272441 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:51.272403 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-054ef1-5b764f8cf-ms552" podUID="070057c0-00c3-49dc-a4d2-868885c1685c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:08:52.204013 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:52.203929 4951 generic.go:356] "Generic (PLEG): container finished" podID="85d10cb3-066c-46fa-a149-209a92a409df" containerID="d539cd44faf50453ea041b22e793b75fd11a246981b674503412043d37ad6a60" exitCode=0 Oct 30 08:08:52.204336 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:52.204222 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" event={"ID":"85d10cb3-066c-46fa-a149-209a92a409df","Type":"ContainerDied","Data":"d539cd44faf50453ea041b22e793b75fd11a246981b674503412043d37ad6a60"} Oct 30 08:08:52.406184 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:52.406159 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 08:08:52.546537 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:52.546495 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-credential-socket\") pod \"85d10cb3-066c-46fa-a149-209a92a409df\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " Oct 30 08:08:52.546747 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:52.546552 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-workload-socket\") pod \"85d10cb3-066c-46fa-a149-209a92a409df\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " Oct 30 08:08:52.546747 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:52.546639 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-workload-certs\") pod \"85d10cb3-066c-46fa-a149-209a92a409df\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " Oct 30 08:08:52.546747 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:52.546695 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-istio-data\") pod \"85d10cb3-066c-46fa-a149-209a92a409df\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " Oct 30 08:08:52.546965 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:52.546803 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85d10cb3-066c-46fa-a149-209a92a409df-istio-token\" (UniqueName: \"kubernetes.io/projected/85d10cb3-066c-46fa-a149-209a92a409df-istio-token\") pod \"85d10cb3-066c-46fa-a149-209a92a409df\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " Oct 30 08:08:52.546965 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:52.546814 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-credential-socket" pod "85d10cb3-066c-46fa-a149-209a92a409df" (UID: "85d10cb3-066c-46fa-a149-209a92a409df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:08:52.546965 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:52.546831 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85d10cb3-066c-46fa-a149-209a92a409df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85d10cb3-066c-46fa-a149-209a92a409df-istiod-ca-cert\") pod \"85d10cb3-066c-46fa-a149-209a92a409df\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " Oct 30 08:08:52.546965 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:52.546927 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85d10cb3-066c-46fa-a149-209a92a409df-kube-api-access-2h5xl\" (UniqueName: \"kubernetes.io/projected/85d10cb3-066c-46fa-a149-209a92a409df-kube-api-access-2h5xl\") pod \"85d10cb3-066c-46fa-a149-209a92a409df\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " Oct 30 08:08:52.546965 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:52.546955 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-istio-envoy\") pod \"85d10cb3-066c-46fa-a149-209a92a409df\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " Oct 30 08:08:52.547251 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:52.546975 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85d10cb3-066c-46fa-a149-209a92a409df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85d10cb3-066c-46fa-a149-209a92a409df-istio-podinfo\") pod \"85d10cb3-066c-46fa-a149-209a92a409df\" (UID: \"85d10cb3-066c-46fa-a149-209a92a409df\") " Oct 30 08:08:52.547251 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:52.546983 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-istio-data" pod "85d10cb3-066c-46fa-a149-209a92a409df" (UID: "85d10cb3-066c-46fa-a149-209a92a409df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:08:52.547251 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:52.547027 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-workload-socket" pod "85d10cb3-066c-46fa-a149-209a92a409df" (UID: "85d10cb3-066c-46fa-a149-209a92a409df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:08:52.547251 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:52.547172 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85d10cb3-066c-46fa-a149-209a92a409df-istiod-ca-cert" pod "85d10cb3-066c-46fa-a149-209a92a409df" (UID: "85d10cb3-066c-46fa-a149-209a92a409df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:08:52.547492 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:52.547354 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-workload-certs" pod "85d10cb3-066c-46fa-a149-209a92a409df" (UID: "85d10cb3-066c-46fa-a149-209a92a409df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:08:52.547492 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:52.547370 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:52.547492 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:52.547390 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:52.547492 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:52.547403 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:52.547492 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:52.547417 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85d10cb3-066c-46fa-a149-209a92a409df-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:52.549339 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:52.549302 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-istio-envoy" pod "85d10cb3-066c-46fa-a149-209a92a409df" (UID: "85d10cb3-066c-46fa-a149-209a92a409df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:08:52.549696 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:52.549669 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85d10cb3-066c-46fa-a149-209a92a409df-kube-api-access-2h5xl" pod "85d10cb3-066c-46fa-a149-209a92a409df" (UID: "85d10cb3-066c-46fa-a149-209a92a409df"). InnerVolumeSpecName "kube-api-access-2h5xl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:52.549800 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:52.549671 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85d10cb3-066c-46fa-a149-209a92a409df-istio-podinfo" pod "85d10cb3-066c-46fa-a149-209a92a409df" (UID: "85d10cb3-066c-46fa-a149-209a92a409df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:08:52.549800 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:52.549781 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85d10cb3-066c-46fa-a149-209a92a409df-istio-token" pod "85d10cb3-066c-46fa-a149-209a92a409df" (UID: "85d10cb3-066c-46fa-a149-209a92a409df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:52.649114 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:52.649062 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:52.649114 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:52.649101 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85d10cb3-066c-46fa-a149-209a92a409df-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:52.649114 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:52.649112 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h5xl\" (UniqueName: \"kubernetes.io/projected/85d10cb3-066c-46fa-a149-209a92a409df-kube-api-access-2h5xl\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:52.649114 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:52.649123 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85d10cb3-066c-46fa-a149-209a92a409df-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:52.649497 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:52.649133 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85d10cb3-066c-46fa-a149-209a92a409df-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:53.024331 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:08:53.024286 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 08:08:53.212176 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:53.212066 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" Oct 30 08:08:53.212176 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:53.212074 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5" event={"ID":"85d10cb3-066c-46fa-a149-209a92a409df","Type":"ContainerDied","Data":"fadfcd78c80bad9c9f09330d59e30a5c2b7fd6ed76c6e000cc3866362a68f4c2"} Oct 30 08:08:53.239996 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:53.212229 4951 scope.go:122] "RemoveContainer" containerID="d539cd44faf50453ea041b22e793b75fd11a246981b674503412043d37ad6a60" Oct 30 08:08:53.277310 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:53.277240 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5"] Oct 30 08:08:53.289815 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:53.289785 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-2rjq5"] Oct 30 08:08:53.781875 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:53.781840 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85d10cb3-066c-46fa-a149-209a92a409df" path="/var/lib/kubelet/pods/85d10cb3-066c-46fa-a149-209a92a409df/volumes" Oct 30 08:08:57.631222 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:57.631190 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-054ef1-5dfd5878db-td8f2" Oct 30 08:08:57.667874 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:57.667847 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-054ef1-5b764f8cf-ms552" Oct 30 08:08:57.725635 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:57.724678 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/070057c0-00c3-49dc-a4d2-868885c1685c-kube-api-access-dt58n\" (UniqueName: \"kubernetes.io/projected/070057c0-00c3-49dc-a4d2-868885c1685c-kube-api-access-dt58n\") pod \"070057c0-00c3-49dc-a4d2-868885c1685c\" (UID: \"070057c0-00c3-49dc-a4d2-868885c1685c\") " Oct 30 08:08:57.725635 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:57.724736 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc-plugins-config-volume\") pod \"8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc\" (UID: \"8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc\") " Oct 30 08:08:57.725635 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:57.724772 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/070057c0-00c3-49dc-a4d2-868885c1685c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/070057c0-00c3-49dc-a4d2-868885c1685c-plugins-config-volume\") pod \"070057c0-00c3-49dc-a4d2-868885c1685c\" (UID: \"070057c0-00c3-49dc-a4d2-868885c1685c\") " Oct 30 08:08:57.725635 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:57.724975 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc-kube-api-access-xjpsk\" (UniqueName: \"kubernetes.io/projected/8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc-kube-api-access-xjpsk\") pod \"8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc\" (UID: \"8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc\") " Oct 30 08:08:57.728719 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:57.728671 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc-plugins-config-volume" pod "8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc" (UID: "8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:08:57.729493 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:57.729469 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/070057c0-00c3-49dc-a4d2-868885c1685c-plugins-config-volume" pod "070057c0-00c3-49dc-a4d2-868885c1685c" (UID: "070057c0-00c3-49dc-a4d2-868885c1685c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:08:57.732694 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:57.732659 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc-kube-api-access-xjpsk" pod "8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc" (UID: "8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc"). InnerVolumeSpecName "kube-api-access-xjpsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:57.733373 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:57.733334 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/070057c0-00c3-49dc-a4d2-868885c1685c-kube-api-access-dt58n" pod "070057c0-00c3-49dc-a4d2-868885c1685c" (UID: "070057c0-00c3-49dc-a4d2-868885c1685c"). InnerVolumeSpecName "kube-api-access-dt58n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:57.804148 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:57.804117 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4eff1f-76974db5db-jtmws"] Oct 30 08:08:57.827018 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:57.826993 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt58n\" (UniqueName: \"kubernetes.io/projected/070057c0-00c3-49dc-a4d2-868885c1685c-kube-api-access-dt58n\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:57.827018 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:57.827018 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:57.827182 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:57.827029 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/070057c0-00c3-49dc-a4d2-868885c1685c-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:57.827182 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:57.827038 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjpsk\" (UniqueName: \"kubernetes.io/projected/8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc-kube-api-access-xjpsk\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:08:57.839657 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:57.839628 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4eff1f-76cb49c85f-vvqsc"] Oct 30 08:08:58.243665 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:58.243623 4951 generic.go:356] "Generic (PLEG): container finished" podID="070057c0-00c3-49dc-a4d2-868885c1685c" containerID="ab330af512d31254a3ea895c288da180a4869d56a8d3976ef5d0698345ce2ea3" exitCode=0 Oct 30 08:08:58.243898 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:58.243677 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-054ef1-5b764f8cf-ms552" Oct 30 08:08:58.243898 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:58.243715 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-054ef1-5b764f8cf-ms552" event={"ID":"070057c0-00c3-49dc-a4d2-868885c1685c","Type":"ContainerDied","Data":"ab330af512d31254a3ea895c288da180a4869d56a8d3976ef5d0698345ce2ea3"} Oct 30 08:08:58.243898 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:58.243783 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-054ef1-5b764f8cf-ms552" event={"ID":"070057c0-00c3-49dc-a4d2-868885c1685c","Type":"ContainerDied","Data":"19ee519847e53e4ab27e8dc8ffdc9b003ca355ea3170857f8b0925d99fb50fb2"} Oct 30 08:08:58.243898 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:58.243817 4951 scope.go:122] "RemoveContainer" containerID="ab330af512d31254a3ea895c288da180a4869d56a8d3976ef5d0698345ce2ea3" Oct 30 08:08:58.245335 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:58.245312 4951 generic.go:356] "Generic (PLEG): container finished" podID="8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc" containerID="12800c6ba21910303e830d18aef96448ad7d17b3f8e26849cd8adc165cde5467" exitCode=0 Oct 30 08:08:58.245471 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:58.245366 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-054ef1-5dfd5878db-td8f2" Oct 30 08:08:58.245471 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:58.245368 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-054ef1-5dfd5878db-td8f2" event={"ID":"8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc","Type":"ContainerDied","Data":"12800c6ba21910303e830d18aef96448ad7d17b3f8e26849cd8adc165cde5467"} Oct 30 08:08:58.245630 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:58.245476 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-054ef1-5dfd5878db-td8f2" event={"ID":"8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc","Type":"ContainerDied","Data":"d9f986e4b8067f1c201db3d6bd0b6efd678dc5d5848279a72d42765a0aae8b96"} Oct 30 08:08:58.253798 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:58.253767 4951 scope.go:122] "RemoveContainer" containerID="ab330af512d31254a3ea895c288da180a4869d56a8d3976ef5d0698345ce2ea3" Oct 30 08:08:58.254172 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:08:58.254149 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab330af512d31254a3ea895c288da180a4869d56a8d3976ef5d0698345ce2ea3\": container with ID starting with ab330af512d31254a3ea895c288da180a4869d56a8d3976ef5d0698345ce2ea3 not found: ID does not exist" containerID="ab330af512d31254a3ea895c288da180a4869d56a8d3976ef5d0698345ce2ea3" Oct 30 08:08:58.254280 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:58.254205 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab330af512d31254a3ea895c288da180a4869d56a8d3976ef5d0698345ce2ea3"} err="failed to get container status \"ab330af512d31254a3ea895c288da180a4869d56a8d3976ef5d0698345ce2ea3\": rpc error: code = NotFound desc = could not find container \"ab330af512d31254a3ea895c288da180a4869d56a8d3976ef5d0698345ce2ea3\": container with ID starting with ab330af512d31254a3ea895c288da180a4869d56a8d3976ef5d0698345ce2ea3 not found: ID does not exist" Oct 30 08:08:58.254280 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:58.254230 4951 scope.go:122] "RemoveContainer" containerID="12800c6ba21910303e830d18aef96448ad7d17b3f8e26849cd8adc165cde5467" Oct 30 08:08:58.263620 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:58.263598 4951 scope.go:122] "RemoveContainer" containerID="12800c6ba21910303e830d18aef96448ad7d17b3f8e26849cd8adc165cde5467" Oct 30 08:08:58.263912 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:08:58.263889 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12800c6ba21910303e830d18aef96448ad7d17b3f8e26849cd8adc165cde5467\": container with ID starting with 12800c6ba21910303e830d18aef96448ad7d17b3f8e26849cd8adc165cde5467 not found: ID does not exist" containerID="12800c6ba21910303e830d18aef96448ad7d17b3f8e26849cd8adc165cde5467" Oct 30 08:08:58.263977 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:58.263921 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12800c6ba21910303e830d18aef96448ad7d17b3f8e26849cd8adc165cde5467"} err="failed to get container status \"12800c6ba21910303e830d18aef96448ad7d17b3f8e26849cd8adc165cde5467\": rpc error: code = NotFound desc = could not find container \"12800c6ba21910303e830d18aef96448ad7d17b3f8e26849cd8adc165cde5467\": container with ID starting with 12800c6ba21910303e830d18aef96448ad7d17b3f8e26849cd8adc165cde5467 not found: ID does not exist" Oct 30 08:08:58.297430 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:58.297392 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-054ef1-5b764f8cf-ms552"] Oct 30 08:08:58.300632 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:58.300602 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-054ef1-5b764f8cf-ms552"] Oct 30 08:08:58.337380 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:58.337352 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-054ef1-5dfd5878db-td8f2"] Oct 30 08:08:58.342468 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:58.342438 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-054ef1-5dfd5878db-td8f2"] Oct 30 08:08:59.772644 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:59.772609 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="070057c0-00c3-49dc-a4d2-868885c1685c" path="/var/lib/kubelet/pods/070057c0-00c3-49dc-a4d2-868885c1685c/volumes" Oct 30 08:08:59.773093 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:08:59.773069 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc" path="/var/lib/kubelet/pods/8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc/volumes" Oct 30 08:09:00.928381 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:00.924458 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4"] Oct 30 08:09:00.928381 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:00.924787 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4" podUID="3f265353-a1b5-4361-bce3-65723e30beec" containerName="maas-api" containerID="cri-o://2b4b5ad2616946f2c02509de072355338f015abc970179950abff7fc291c7633" gracePeriod=30 Oct 30 08:09:01.209225 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:01.209157 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4" Oct 30 08:09:01.267447 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:01.266582 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f265353-a1b5-4361-bce3-65723e30beec-kube-api-access-l8ql4\" (UniqueName: \"kubernetes.io/projected/3f265353-a1b5-4361-bce3-65723e30beec-kube-api-access-l8ql4\") pod \"3f265353-a1b5-4361-bce3-65723e30beec\" (UID: \"3f265353-a1b5-4361-bce3-65723e30beec\") " Oct 30 08:09:01.267447 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:01.266636 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f265353-a1b5-4361-bce3-65723e30beec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f265353-a1b5-4361-bce3-65723e30beec-maas-api-tls\") pod \"3f265353-a1b5-4361-bce3-65723e30beec\" (UID: \"3f265353-a1b5-4361-bce3-65723e30beec\") " Oct 30 08:09:01.267447 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:01.266679 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f265353-a1b5-4361-bce3-65723e30beec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f265353-a1b5-4361-bce3-65723e30beec-metrics-certs\") pod \"3f265353-a1b5-4361-bce3-65723e30beec\" (UID: \"3f265353-a1b5-4361-bce3-65723e30beec\") " Oct 30 08:09:01.267447 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:01.266786 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f265353-a1b5-4361-bce3-65723e30beec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f265353-a1b5-4361-bce3-65723e30beec-openshift-service-ca\") pod \"3f265353-a1b5-4361-bce3-65723e30beec\" (UID: \"3f265353-a1b5-4361-bce3-65723e30beec\") " Oct 30 08:09:01.268339 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:01.268311 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f265353-a1b5-4361-bce3-65723e30beec-openshift-service-ca" pod "3f265353-a1b5-4361-bce3-65723e30beec" (UID: "3f265353-a1b5-4361-bce3-65723e30beec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:09:01.272531 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:01.271516 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f265353-a1b5-4361-bce3-65723e30beec-maas-api-tls" pod "3f265353-a1b5-4361-bce3-65723e30beec" (UID: "3f265353-a1b5-4361-bce3-65723e30beec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:09:01.272721 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:01.271538 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f265353-a1b5-4361-bce3-65723e30beec-metrics-certs" pod "3f265353-a1b5-4361-bce3-65723e30beec" (UID: "3f265353-a1b5-4361-bce3-65723e30beec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:09:01.272927 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:01.272902 4951 generic.go:356] "Generic (PLEG): container finished" podID="3f265353-a1b5-4361-bce3-65723e30beec" containerID="2b4b5ad2616946f2c02509de072355338f015abc970179950abff7fc291c7633" exitCode=0 Oct 30 08:09:01.273027 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:01.272982 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4" event={"ID":"3f265353-a1b5-4361-bce3-65723e30beec","Type":"ContainerDied","Data":"2b4b5ad2616946f2c02509de072355338f015abc970179950abff7fc291c7633"} Oct 30 08:09:01.273102 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:01.273034 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4" event={"ID":"3f265353-a1b5-4361-bce3-65723e30beec","Type":"ContainerDied","Data":"0a70d69d73adbd86536eaaacbc34b6387270194b25fe61886eca82d847d7b6cf"} Oct 30 08:09:01.273102 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:01.273072 4951 scope.go:122] "RemoveContainer" containerID="2b4b5ad2616946f2c02509de072355338f015abc970179950abff7fc291c7633" Oct 30 08:09:01.273252 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:01.273236 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4" Oct 30 08:09:01.275619 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:01.275554 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f265353-a1b5-4361-bce3-65723e30beec-kube-api-access-l8ql4" pod "3f265353-a1b5-4361-bce3-65723e30beec" (UID: "3f265353-a1b5-4361-bce3-65723e30beec"). InnerVolumeSpecName "kube-api-access-l8ql4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:01.296178 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:01.296155 4951 scope.go:122] "RemoveContainer" containerID="2b4b5ad2616946f2c02509de072355338f015abc970179950abff7fc291c7633" Oct 30 08:09:01.296509 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:09:01.296483 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b4b5ad2616946f2c02509de072355338f015abc970179950abff7fc291c7633\": container with ID starting with 2b4b5ad2616946f2c02509de072355338f015abc970179950abff7fc291c7633 not found: ID does not exist" containerID="2b4b5ad2616946f2c02509de072355338f015abc970179950abff7fc291c7633" Oct 30 08:09:01.296595 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:01.296517 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b4b5ad2616946f2c02509de072355338f015abc970179950abff7fc291c7633"} err="failed to get container status \"2b4b5ad2616946f2c02509de072355338f015abc970179950abff7fc291c7633\": rpc error: code = NotFound desc = could not find container \"2b4b5ad2616946f2c02509de072355338f015abc970179950abff7fc291c7633\": container with ID starting with 2b4b5ad2616946f2c02509de072355338f015abc970179950abff7fc291c7633 not found: ID does not exist" Oct 30 08:09:01.368074 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:01.368036 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8ql4\" (UniqueName: \"kubernetes.io/projected/3f265353-a1b5-4361-bce3-65723e30beec-kube-api-access-l8ql4\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:01.368074 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:01.368068 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f265353-a1b5-4361-bce3-65723e30beec-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:01.368074 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:01.368078 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f265353-a1b5-4361-bce3-65723e30beec-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:01.368402 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:01.368086 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f265353-a1b5-4361-bce3-65723e30beec-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:01.623264 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:01.623230 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4"] Oct 30 08:09:01.636764 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:01.636717 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4eff1f-55c9577495-hj8l4"] Oct 30 08:09:01.773386 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:01.773352 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f265353-a1b5-4361-bce3-65723e30beec" path="/var/lib/kubelet/pods/3f265353-a1b5-4361-bce3-65723e30beec/volumes" Oct 30 08:09:02.804746 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:02.804697 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4eff1f-76974db5db-jtmws" podUID="325e2932-31f7-49a9-bac7-d8ac85a4500f" containerName="payload-processing" containerID="cri-o://d3a98c02056dade6e0fe7b9893c4c99d73d5a05b82ed88826f4304a4827590b4" gracePeriod=25 Oct 30 08:09:02.840064 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:02.840027 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4eff1f-76cb49c85f-vvqsc" podUID="1016f833-117b-4e7a-9678-5dc7b3656bfe" containerName="payload-pre-processing" containerID="cri-o://5983b8f0bf5e6d7aec6cf088edd94c1ce059626ead3977cbe7dc7075e21457c3" gracePeriod=25 Oct 30 08:09:04.244896 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.244853 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" containerID="cri-o://ef6ab34e6e1b3c8dcd372d5340fd5d0f266a6fc7fe4d917cb42b96a27938bcdc" gracePeriod=45 Oct 30 08:09:04.274232 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.274198 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" containerID="cri-o://7d073a0f59654b3d4dcef450f56fbb90ee73060b644a13a569407a878f68635e" gracePeriod=45 Oct 30 08:09:04.554233 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.554207 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 08:09:04.569123 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.569104 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 08:09:04.704240 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.704211 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-home\" (UniqueName: \"kubernetes.io/empty-dir/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-home\") pod \"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106\" (UID: \"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106\") " Oct 30 08:09:04.704437 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.704255 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-dshm\") pod \"e4c39d5b-24e5-433d-a50f-b7e5f45b993f\" (UID: \"e4c39d5b-24e5-433d-a50f-b7e5f45b993f\") " Oct 30 08:09:04.704437 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.704281 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-tls-certs\") pod \"e4c39d5b-24e5-433d-a50f-b7e5f45b993f\" (UID: \"e4c39d5b-24e5-433d-a50f-b7e5f45b993f\") " Oct 30 08:09:04.704437 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.704335 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-tmp-dir\") pod \"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106\" (UID: \"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106\") " Oct 30 08:09:04.704437 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.704362 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-dshm\") pod \"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106\" (UID: \"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106\") " Oct 30 08:09:04.704437 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.704400 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-model-cache\") pod \"e4c39d5b-24e5-433d-a50f-b7e5f45b993f\" (UID: \"e4c39d5b-24e5-433d-a50f-b7e5f45b993f\") " Oct 30 08:09:04.704763 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.704499 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-kube-api-access-vm6sf\" (UniqueName: \"kubernetes.io/projected/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-kube-api-access-vm6sf\") pod \"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106\" (UID: \"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106\") " Oct 30 08:09:04.704763 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.704506 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-home" pod "8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" (UID: "8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:04.704763 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.704539 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-tmp-dir\") pod \"e4c39d5b-24e5-433d-a50f-b7e5f45b993f\" (UID: \"e4c39d5b-24e5-433d-a50f-b7e5f45b993f\") " Oct 30 08:09:04.704763 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.704600 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-model-cache\") pod \"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106\" (UID: \"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106\") " Oct 30 08:09:04.704763 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.704642 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-tls-certs\" (UniqueName: \"kubernetes.io/secret/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-tls-certs\") pod \"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106\" (UID: \"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106\") " Oct 30 08:09:04.704763 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.704732 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-model-cache" pod "e4c39d5b-24e5-433d-a50f-b7e5f45b993f" (UID: "e4c39d5b-24e5-433d-a50f-b7e5f45b993f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:04.705101 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.704762 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-home\" (UniqueName: \"kubernetes.io/empty-dir/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-home\") pod \"e4c39d5b-24e5-433d-a50f-b7e5f45b993f\" (UID: \"e4c39d5b-24e5-433d-a50f-b7e5f45b993f\") " Oct 30 08:09:04.705101 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.704809 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-kube-api-access-cvd7l\" (UniqueName: \"kubernetes.io/projected/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-kube-api-access-cvd7l\") pod \"e4c39d5b-24e5-433d-a50f-b7e5f45b993f\" (UID: \"e4c39d5b-24e5-433d-a50f-b7e5f45b993f\") " Oct 30 08:09:04.705251 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.705232 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-model-cache" pod "8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" (UID: "8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:04.705330 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.705299 4951 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-home\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:04.705330 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.705319 4951 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-model-cache\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:04.705477 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.705340 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-home" pod "e4c39d5b-24e5-433d-a50f-b7e5f45b993f" (UID: "e4c39d5b-24e5-433d-a50f-b7e5f45b993f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:04.705545 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.705522 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-tmp-dir" pod "e4c39d5b-24e5-433d-a50f-b7e5f45b993f" (UID: "e4c39d5b-24e5-433d-a50f-b7e5f45b993f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:04.705631 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.705607 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-tmp-dir" pod "8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" (UID: "8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:04.708000 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.707964 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-kube-api-access-cvd7l" pod "e4c39d5b-24e5-433d-a50f-b7e5f45b993f" (UID: "e4c39d5b-24e5-433d-a50f-b7e5f45b993f"). InnerVolumeSpecName "kube-api-access-cvd7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:04.708136 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.707994 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-tls-certs" pod "8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" (UID: "8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:09:04.708136 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.708017 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-dshm" pod "8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" (UID: "8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:04.708136 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.708033 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-kube-api-access-vm6sf" pod "8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" (UID: "8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106"). InnerVolumeSpecName "kube-api-access-vm6sf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:04.708335 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.708167 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-tls-certs" pod "e4c39d5b-24e5-433d-a50f-b7e5f45b993f" (UID: "e4c39d5b-24e5-433d-a50f-b7e5f45b993f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:09:04.709407 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.709386 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-dshm" pod "e4c39d5b-24e5-433d-a50f-b7e5f45b993f" (UID: "e4c39d5b-24e5-433d-a50f-b7e5f45b993f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:04.805870 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.805788 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm6sf\" (UniqueName: \"kubernetes.io/projected/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-kube-api-access-vm6sf\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:04.805870 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.805816 4951 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-tmp-dir\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:04.805870 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.805828 4951 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-model-cache\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:04.805870 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.805837 4951 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-tls-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:04.805870 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.805845 4951 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-home\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:04.805870 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.805854 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvd7l\" (UniqueName: \"kubernetes.io/projected/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-kube-api-access-cvd7l\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:04.805870 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.805862 4951 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-dshm\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:04.805870 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.805869 4951 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c39d5b-24e5-433d-a50f-b7e5f45b993f-tls-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:04.805870 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.805877 4951 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-tmp-dir\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:04.806485 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:04.805885 4951 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106-dshm\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:05.299939 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:05.299904 4951 generic.go:356] "Generic (PLEG): container finished" podID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerID="ef6ab34e6e1b3c8dcd372d5340fd5d0f266a6fc7fe4d917cb42b96a27938bcdc" exitCode=0 Oct 30 08:09:05.300460 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:05.299966 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" Oct 30 08:09:05.300460 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:05.299990 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" event={"ID":"e4c39d5b-24e5-433d-a50f-b7e5f45b993f","Type":"ContainerDied","Data":"ef6ab34e6e1b3c8dcd372d5340fd5d0f266a6fc7fe4d917cb42b96a27938bcdc"} Oct 30 08:09:05.300460 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:05.300038 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" event={"ID":"e4c39d5b-24e5-433d-a50f-b7e5f45b993f","Type":"ContainerDied","Data":"6bd4ddb7a1e6e14cf3b6bd8315d7789db361fcb606f28314f06a1e1407cc8220"} Oct 30 08:09:05.300460 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:05.300060 4951 scope.go:122] "RemoveContainer" containerID="ef6ab34e6e1b3c8dcd372d5340fd5d0f266a6fc7fe4d917cb42b96a27938bcdc" Oct 30 08:09:05.301661 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:05.301558 4951 generic.go:356] "Generic (PLEG): container finished" podID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerID="7d073a0f59654b3d4dcef450f56fbb90ee73060b644a13a569407a878f68635e" exitCode=0 Oct 30 08:09:05.301661 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:05.301619 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" event={"ID":"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106","Type":"ContainerDied","Data":"7d073a0f59654b3d4dcef450f56fbb90ee73060b644a13a569407a878f68635e"} Oct 30 08:09:05.301814 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:05.301671 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" event={"ID":"8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106","Type":"ContainerDied","Data":"035c8e774135ed0908771cbc23f0ee6bc31431cf50812aa5acbb0693a0bd6d53"} Oct 30 08:09:05.301814 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:05.301674 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr" Oct 30 08:09:05.319397 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:05.319372 4951 scope.go:122] "RemoveContainer" containerID="ef6ab34e6e1b3c8dcd372d5340fd5d0f266a6fc7fe4d917cb42b96a27938bcdc" Oct 30 08:09:05.319795 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:09:05.319776 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef6ab34e6e1b3c8dcd372d5340fd5d0f266a6fc7fe4d917cb42b96a27938bcdc\": container with ID starting with ef6ab34e6e1b3c8dcd372d5340fd5d0f266a6fc7fe4d917cb42b96a27938bcdc not found: ID does not exist" containerID="ef6ab34e6e1b3c8dcd372d5340fd5d0f266a6fc7fe4d917cb42b96a27938bcdc" Oct 30 08:09:05.319864 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:05.319805 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef6ab34e6e1b3c8dcd372d5340fd5d0f266a6fc7fe4d917cb42b96a27938bcdc"} err="failed to get container status \"ef6ab34e6e1b3c8dcd372d5340fd5d0f266a6fc7fe4d917cb42b96a27938bcdc\": rpc error: code = NotFound desc = could not find container \"ef6ab34e6e1b3c8dcd372d5340fd5d0f266a6fc7fe4d917cb42b96a27938bcdc\": container with ID starting with ef6ab34e6e1b3c8dcd372d5340fd5d0f266a6fc7fe4d917cb42b96a27938bcdc not found: ID does not exist" Oct 30 08:09:05.319864 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:05.319822 4951 scope.go:122] "RemoveContainer" containerID="7d073a0f59654b3d4dcef450f56fbb90ee73060b644a13a569407a878f68635e" Oct 30 08:09:05.331462 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:05.331438 4951 scope.go:122] "RemoveContainer" containerID="7d073a0f59654b3d4dcef450f56fbb90ee73060b644a13a569407a878f68635e" Oct 30 08:09:05.331856 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:09:05.331834 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d073a0f59654b3d4dcef450f56fbb90ee73060b644a13a569407a878f68635e\": container with ID starting with 7d073a0f59654b3d4dcef450f56fbb90ee73060b644a13a569407a878f68635e not found: ID does not exist" containerID="7d073a0f59654b3d4dcef450f56fbb90ee73060b644a13a569407a878f68635e" Oct 30 08:09:05.331933 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:05.331866 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d073a0f59654b3d4dcef450f56fbb90ee73060b644a13a569407a878f68635e"} err="failed to get container status \"7d073a0f59654b3d4dcef450f56fbb90ee73060b644a13a569407a878f68635e\": rpc error: code = NotFound desc = could not find container \"7d073a0f59654b3d4dcef450f56fbb90ee73060b644a13a569407a878f68635e\": container with ID starting with 7d073a0f59654b3d4dcef450f56fbb90ee73060b644a13a569407a878f68635e not found: ID does not exist" Oct 30 08:09:05.359401 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:05.359372 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2"] Oct 30 08:09:05.373234 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:05.373202 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2"] Oct 30 08:09:05.402611 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:05.402565 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr"] Oct 30 08:09:05.407655 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:05.407623 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/simulator-w1-a0b3f5-kserve-767968f6cc-nklmr"] Oct 30 08:09:05.503934 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:05.503885 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-2-w1-a0b3f5-kserve-8847d7b5b-vctl2" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 08:09:05.771784 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:05.771746 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" path="/var/lib/kubelet/pods/8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106/volumes" Oct 30 08:09:05.772236 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:05.772220 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" path="/var/lib/kubelet/pods/e4c39d5b-24e5-433d-a50f-b7e5f45b993f/volumes" Oct 30 08:09:07.847382 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:07.847346 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4eff1f-76cb49c85f-vvqsc" podUID="1016f833-117b-4e7a-9678-5dc7b3656bfe" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:09:08.853701 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:08.853664 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4eff1f-76974db5db-jtmws" podUID="325e2932-31f7-49a9-bac7-d8ac85a4500f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:09:08.975088 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:08.975047 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr"] Oct 30 08:09:08.975313 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:08.975289 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" podUID="db4df589-43fd-4413-8c51-de66dd1ec2e2" containerName="istio-proxy" containerID="cri-o://d58458f3034fc77d477167f50303b7e144f2e9cc1028259f475a7df1baaf079c" gracePeriod=30 Oct 30 08:09:12.376364 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:12.376330 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t"] Oct 30 08:09:12.376886 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:12.376675 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t" podUID="910285f4-8b96-44a7-a0e5-542f04538151" containerName="maas-api" containerID="cri-o://bce973ee83c8fcb6e8b525b7478edbcf1a5b595e4c63b85cd0ea6c6a3f98b155" gracePeriod=30 Oct 30 08:09:12.650070 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:12.650042 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t" Oct 30 08:09:12.806975 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:12.806937 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/910285f4-8b96-44a7-a0e5-542f04538151-metrics-certs\" (UniqueName: \"kubernetes.io/secret/910285f4-8b96-44a7-a0e5-542f04538151-metrics-certs\") pod \"910285f4-8b96-44a7-a0e5-542f04538151\" (UID: \"910285f4-8b96-44a7-a0e5-542f04538151\") " Oct 30 08:09:12.807194 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:12.806989 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/910285f4-8b96-44a7-a0e5-542f04538151-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/910285f4-8b96-44a7-a0e5-542f04538151-maas-api-tls\") pod \"910285f4-8b96-44a7-a0e5-542f04538151\" (UID: \"910285f4-8b96-44a7-a0e5-542f04538151\") " Oct 30 08:09:12.807194 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:12.807018 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/910285f4-8b96-44a7-a0e5-542f04538151-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/910285f4-8b96-44a7-a0e5-542f04538151-openshift-service-ca\") pod \"910285f4-8b96-44a7-a0e5-542f04538151\" (UID: \"910285f4-8b96-44a7-a0e5-542f04538151\") " Oct 30 08:09:12.807194 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:12.807044 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/910285f4-8b96-44a7-a0e5-542f04538151-kube-api-access-jk7bk\" (UniqueName: \"kubernetes.io/projected/910285f4-8b96-44a7-a0e5-542f04538151-kube-api-access-jk7bk\") pod \"910285f4-8b96-44a7-a0e5-542f04538151\" (UID: \"910285f4-8b96-44a7-a0e5-542f04538151\") " Oct 30 08:09:12.807470 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:12.807438 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/910285f4-8b96-44a7-a0e5-542f04538151-openshift-service-ca" pod "910285f4-8b96-44a7-a0e5-542f04538151" (UID: "910285f4-8b96-44a7-a0e5-542f04538151"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:09:12.809348 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:12.809309 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/910285f4-8b96-44a7-a0e5-542f04538151-kube-api-access-jk7bk" pod "910285f4-8b96-44a7-a0e5-542f04538151" (UID: "910285f4-8b96-44a7-a0e5-542f04538151"). InnerVolumeSpecName "kube-api-access-jk7bk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:12.809691 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:12.809666 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/910285f4-8b96-44a7-a0e5-542f04538151-maas-api-tls" pod "910285f4-8b96-44a7-a0e5-542f04538151" (UID: "910285f4-8b96-44a7-a0e5-542f04538151"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:09:12.809691 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:12.809680 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/910285f4-8b96-44a7-a0e5-542f04538151-metrics-certs" pod "910285f4-8b96-44a7-a0e5-542f04538151" (UID: "910285f4-8b96-44a7-a0e5-542f04538151"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:09:12.909084 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:12.908979 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/910285f4-8b96-44a7-a0e5-542f04538151-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:12.909084 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:12.909009 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/910285f4-8b96-44a7-a0e5-542f04538151-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:12.909084 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:12.909018 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/910285f4-8b96-44a7-a0e5-542f04538151-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:12.909084 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:12.909026 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk7bk\" (UniqueName: \"kubernetes.io/projected/910285f4-8b96-44a7-a0e5-542f04538151-kube-api-access-jk7bk\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:13.349960 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:13.349925 4951 generic.go:356] "Generic (PLEG): container finished" podID="910285f4-8b96-44a7-a0e5-542f04538151" containerID="bce973ee83c8fcb6e8b525b7478edbcf1a5b595e4c63b85cd0ea6c6a3f98b155" exitCode=0 Oct 30 08:09:13.350177 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:13.349975 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t" Oct 30 08:09:13.350177 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:13.349986 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t" event={"ID":"910285f4-8b96-44a7-a0e5-542f04538151","Type":"ContainerDied","Data":"bce973ee83c8fcb6e8b525b7478edbcf1a5b595e4c63b85cd0ea6c6a3f98b155"} Oct 30 08:09:13.350177 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:13.350036 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t" event={"ID":"910285f4-8b96-44a7-a0e5-542f04538151","Type":"ContainerDied","Data":"399b1319b13c1c7f00adbd141736f92ad070f77d178dbc2f1a5c218d97aeec6f"} Oct 30 08:09:13.350177 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:13.350063 4951 scope.go:122] "RemoveContainer" containerID="bce973ee83c8fcb6e8b525b7478edbcf1a5b595e4c63b85cd0ea6c6a3f98b155" Oct 30 08:09:13.359856 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:13.359838 4951 scope.go:122] "RemoveContainer" containerID="bce973ee83c8fcb6e8b525b7478edbcf1a5b595e4c63b85cd0ea6c6a3f98b155" Oct 30 08:09:13.360135 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:09:13.360117 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bce973ee83c8fcb6e8b525b7478edbcf1a5b595e4c63b85cd0ea6c6a3f98b155\": container with ID starting with bce973ee83c8fcb6e8b525b7478edbcf1a5b595e4c63b85cd0ea6c6a3f98b155 not found: ID does not exist" containerID="bce973ee83c8fcb6e8b525b7478edbcf1a5b595e4c63b85cd0ea6c6a3f98b155" Oct 30 08:09:13.360187 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:13.360145 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bce973ee83c8fcb6e8b525b7478edbcf1a5b595e4c63b85cd0ea6c6a3f98b155"} err="failed to get container status \"bce973ee83c8fcb6e8b525b7478edbcf1a5b595e4c63b85cd0ea6c6a3f98b155\": rpc error: code = NotFound desc = could not find container \"bce973ee83c8fcb6e8b525b7478edbcf1a5b595e4c63b85cd0ea6c6a3f98b155\": container with ID starting with bce973ee83c8fcb6e8b525b7478edbcf1a5b595e4c63b85cd0ea6c6a3f98b155 not found: ID does not exist" Oct 30 08:09:13.403412 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:13.403383 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t"] Oct 30 08:09:13.415932 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:13.415908 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bd38cf-5bd854cc5d-fjr7t"] Oct 30 08:09:13.770388 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:13.770340 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="910285f4-8b96-44a7-a0e5-542f04538151" path="/var/lib/kubelet/pods/910285f4-8b96-44a7-a0e5-542f04538151/volumes" Oct 30 08:09:14.224649 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.224617 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 08:09:14.324452 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.324418 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db4df589-43fd-4413-8c51-de66dd1ec2e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db4df589-43fd-4413-8c51-de66dd1ec2e2-istio-podinfo\") pod \"db4df589-43fd-4413-8c51-de66dd1ec2e2\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " Oct 30 08:09:14.324663 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.324488 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-istio-envoy\") pod \"db4df589-43fd-4413-8c51-de66dd1ec2e2\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " Oct 30 08:09:14.324663 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.324522 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-istio-data\") pod \"db4df589-43fd-4413-8c51-de66dd1ec2e2\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " Oct 30 08:09:14.324663 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.324544 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-workload-certs\") pod \"db4df589-43fd-4413-8c51-de66dd1ec2e2\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " Oct 30 08:09:14.324663 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.324561 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db4df589-43fd-4413-8c51-de66dd1ec2e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db4df589-43fd-4413-8c51-de66dd1ec2e2-istiod-ca-cert\") pod \"db4df589-43fd-4413-8c51-de66dd1ec2e2\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " Oct 30 08:09:14.324663 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.324609 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db4df589-43fd-4413-8c51-de66dd1ec2e2-kube-api-access-6g4hh\" (UniqueName: \"kubernetes.io/projected/db4df589-43fd-4413-8c51-de66dd1ec2e2-kube-api-access-6g4hh\") pod \"db4df589-43fd-4413-8c51-de66dd1ec2e2\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " Oct 30 08:09:14.324998 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.324684 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-credential-socket\") pod \"db4df589-43fd-4413-8c51-de66dd1ec2e2\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " Oct 30 08:09:14.324998 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.324746 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-workload-socket\") pod \"db4df589-43fd-4413-8c51-de66dd1ec2e2\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " Oct 30 08:09:14.324998 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.324847 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db4df589-43fd-4413-8c51-de66dd1ec2e2-istio-token\" (UniqueName: \"kubernetes.io/projected/db4df589-43fd-4413-8c51-de66dd1ec2e2-istio-token\") pod \"db4df589-43fd-4413-8c51-de66dd1ec2e2\" (UID: \"db4df589-43fd-4413-8c51-de66dd1ec2e2\") " Oct 30 08:09:14.324998 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.324866 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-istio-data" pod "db4df589-43fd-4413-8c51-de66dd1ec2e2" (UID: "db4df589-43fd-4413-8c51-de66dd1ec2e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:14.324998 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.324896 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-workload-certs" pod "db4df589-43fd-4413-8c51-de66dd1ec2e2" (UID: "db4df589-43fd-4413-8c51-de66dd1ec2e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:14.325280 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.325015 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-credential-socket" pod "db4df589-43fd-4413-8c51-de66dd1ec2e2" (UID: "db4df589-43fd-4413-8c51-de66dd1ec2e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:14.325280 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.325098 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-workload-socket" pod "db4df589-43fd-4413-8c51-de66dd1ec2e2" (UID: "db4df589-43fd-4413-8c51-de66dd1ec2e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:14.325280 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.325243 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:14.325280 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.325264 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:14.325280 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.325270 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db4df589-43fd-4413-8c51-de66dd1ec2e2-istiod-ca-cert" pod "db4df589-43fd-4413-8c51-de66dd1ec2e2" (UID: "db4df589-43fd-4413-8c51-de66dd1ec2e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:09:14.325589 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.325277 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:14.325589 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.325316 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:14.326907 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.326876 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db4df589-43fd-4413-8c51-de66dd1ec2e2-istio-podinfo" pod "db4df589-43fd-4413-8c51-de66dd1ec2e2" (UID: "db4df589-43fd-4413-8c51-de66dd1ec2e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:09:14.327059 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.326982 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-istio-envoy" pod "db4df589-43fd-4413-8c51-de66dd1ec2e2" (UID: "db4df589-43fd-4413-8c51-de66dd1ec2e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:14.327290 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.327261 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db4df589-43fd-4413-8c51-de66dd1ec2e2-kube-api-access-6g4hh" pod "db4df589-43fd-4413-8c51-de66dd1ec2e2" (UID: "db4df589-43fd-4413-8c51-de66dd1ec2e2"). InnerVolumeSpecName "kube-api-access-6g4hh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:14.327384 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.327322 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db4df589-43fd-4413-8c51-de66dd1ec2e2-istio-token" pod "db4df589-43fd-4413-8c51-de66dd1ec2e2" (UID: "db4df589-43fd-4413-8c51-de66dd1ec2e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:14.357340 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.357281 4951 generic.go:356] "Generic (PLEG): container finished" podID="db4df589-43fd-4413-8c51-de66dd1ec2e2" containerID="d58458f3034fc77d477167f50303b7e144f2e9cc1028259f475a7df1baaf079c" exitCode=0 Oct 30 08:09:14.357429 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.357342 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" Oct 30 08:09:14.357429 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.357367 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" event={"ID":"db4df589-43fd-4413-8c51-de66dd1ec2e2","Type":"ContainerDied","Data":"d58458f3034fc77d477167f50303b7e144f2e9cc1028259f475a7df1baaf079c"} Oct 30 08:09:14.357525 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.357429 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr" event={"ID":"db4df589-43fd-4413-8c51-de66dd1ec2e2","Type":"ContainerDied","Data":"6d419dc73c274fc2641dc02d5ffe48d573a391bec27b3400717a761b1c35590d"} Oct 30 08:09:14.357525 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.357457 4951 scope.go:122] "RemoveContainer" containerID="d58458f3034fc77d477167f50303b7e144f2e9cc1028259f475a7df1baaf079c" Oct 30 08:09:14.366975 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.366958 4951 scope.go:122] "RemoveContainer" containerID="8398656a459109c55a08849f8661b4625027096103bbc31acefc58a5c6273516" Oct 30 08:09:14.378060 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.378037 4951 scope.go:122] "RemoveContainer" containerID="d58458f3034fc77d477167f50303b7e144f2e9cc1028259f475a7df1baaf079c" Oct 30 08:09:14.378489 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:09:14.378461 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d58458f3034fc77d477167f50303b7e144f2e9cc1028259f475a7df1baaf079c\": container with ID starting with d58458f3034fc77d477167f50303b7e144f2e9cc1028259f475a7df1baaf079c not found: ID does not exist" containerID="d58458f3034fc77d477167f50303b7e144f2e9cc1028259f475a7df1baaf079c" Oct 30 08:09:14.378629 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.378495 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d58458f3034fc77d477167f50303b7e144f2e9cc1028259f475a7df1baaf079c"} err="failed to get container status \"d58458f3034fc77d477167f50303b7e144f2e9cc1028259f475a7df1baaf079c\": rpc error: code = NotFound desc = could not find container \"d58458f3034fc77d477167f50303b7e144f2e9cc1028259f475a7df1baaf079c\": container with ID starting with d58458f3034fc77d477167f50303b7e144f2e9cc1028259f475a7df1baaf079c not found: ID does not exist" Oct 30 08:09:14.378629 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.378513 4951 scope.go:122] "RemoveContainer" containerID="8398656a459109c55a08849f8661b4625027096103bbc31acefc58a5c6273516" Oct 30 08:09:14.378822 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:09:14.378799 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8398656a459109c55a08849f8661b4625027096103bbc31acefc58a5c6273516\": container with ID starting with 8398656a459109c55a08849f8661b4625027096103bbc31acefc58a5c6273516 not found: ID does not exist" containerID="8398656a459109c55a08849f8661b4625027096103bbc31acefc58a5c6273516" Oct 30 08:09:14.378868 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.378826 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8398656a459109c55a08849f8661b4625027096103bbc31acefc58a5c6273516"} err="failed to get container status \"8398656a459109c55a08849f8661b4625027096103bbc31acefc58a5c6273516\": rpc error: code = NotFound desc = could not find container \"8398656a459109c55a08849f8661b4625027096103bbc31acefc58a5c6273516\": container with ID starting with 8398656a459109c55a08849f8661b4625027096103bbc31acefc58a5c6273516 not found: ID does not exist" Oct 30 08:09:14.409971 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.409944 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792"] Oct 30 08:09:14.423931 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.423904 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr"] Oct 30 08:09:14.426872 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.426839 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db4df589-43fd-4413-8c51-de66dd1ec2e2-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:14.426872 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.426869 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db4df589-43fd-4413-8c51-de66dd1ec2e2-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:14.427047 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.426882 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db4df589-43fd-4413-8c51-de66dd1ec2e2-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:14.427047 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.426897 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db4df589-43fd-4413-8c51-de66dd1ec2e2-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:14.427047 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.426910 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g4hh\" (UniqueName: \"kubernetes.io/projected/db4df589-43fd-4413-8c51-de66dd1ec2e2-kube-api-access-6g4hh\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:14.429626 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:14.429595 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-zs4kr"] Oct 30 08:09:15.770471 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:15.770392 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db4df589-43fd-4413-8c51-de66dd1ec2e2" path="/var/lib/kubelet/pods/db4df589-43fd-4413-8c51-de66dd1ec2e2/volumes" Oct 30 08:09:17.848311 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:17.848267 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4eff1f-76cb49c85f-vvqsc" podUID="1016f833-117b-4e7a-9678-5dc7b3656bfe" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:09:18.853335 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:18.853295 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4eff1f-76974db5db-jtmws" podUID="325e2932-31f7-49a9-bac7-d8ac85a4500f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:09:20.159589 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:20.159539 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e5fa2d-6dd56c7bf6-96xb9"] Oct 30 08:09:21.624427 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:21.624387 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j"] Oct 30 08:09:21.624978 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:21.624710 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" podUID="a796d7df-446c-419e-be91-f479e9bfa8e2" containerName="istio-proxy" containerID="cri-o://6bacce6fd9a94862334d7f540bf1296875d0445f548e36b69c7cab628ce7f2ef" gracePeriod=30 Oct 30 08:09:23.056855 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:09:23.056659 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 08:09:23.129629 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.129559 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4eff1f-76cb49c85f-vvqsc" Oct 30 08:09:23.164637 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.164601 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4eff1f-76974db5db-jtmws" Oct 30 08:09:23.220598 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.220537 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/325e2932-31f7-49a9-bac7-d8ac85a4500f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/325e2932-31f7-49a9-bac7-d8ac85a4500f-plugins-config-volume\") pod \"325e2932-31f7-49a9-bac7-d8ac85a4500f\" (UID: \"325e2932-31f7-49a9-bac7-d8ac85a4500f\") " Oct 30 08:09:23.220842 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.220650 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1016f833-117b-4e7a-9678-5dc7b3656bfe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1016f833-117b-4e7a-9678-5dc7b3656bfe-plugins-config-volume\") pod \"1016f833-117b-4e7a-9678-5dc7b3656bfe\" (UID: \"1016f833-117b-4e7a-9678-5dc7b3656bfe\") " Oct 30 08:09:23.220842 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.220685 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1016f833-117b-4e7a-9678-5dc7b3656bfe-kube-api-access-5mqs9\" (UniqueName: \"kubernetes.io/projected/1016f833-117b-4e7a-9678-5dc7b3656bfe-kube-api-access-5mqs9\") pod \"1016f833-117b-4e7a-9678-5dc7b3656bfe\" (UID: \"1016f833-117b-4e7a-9678-5dc7b3656bfe\") " Oct 30 08:09:23.220842 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.220733 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/325e2932-31f7-49a9-bac7-d8ac85a4500f-kube-api-access-jh9wt\" (UniqueName: \"kubernetes.io/projected/325e2932-31f7-49a9-bac7-d8ac85a4500f-kube-api-access-jh9wt\") pod \"325e2932-31f7-49a9-bac7-d8ac85a4500f\" (UID: \"325e2932-31f7-49a9-bac7-d8ac85a4500f\") " Oct 30 08:09:23.221032 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.220944 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/325e2932-31f7-49a9-bac7-d8ac85a4500f-plugins-config-volume" pod "325e2932-31f7-49a9-bac7-d8ac85a4500f" (UID: "325e2932-31f7-49a9-bac7-d8ac85a4500f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:09:23.221032 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.221006 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1016f833-117b-4e7a-9678-5dc7b3656bfe-plugins-config-volume" pod "1016f833-117b-4e7a-9678-5dc7b3656bfe" (UID: "1016f833-117b-4e7a-9678-5dc7b3656bfe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:09:23.221366 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.221109 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/325e2932-31f7-49a9-bac7-d8ac85a4500f-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:23.221366 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.221124 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1016f833-117b-4e7a-9678-5dc7b3656bfe-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:23.223098 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.223072 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1016f833-117b-4e7a-9678-5dc7b3656bfe-kube-api-access-5mqs9" pod "1016f833-117b-4e7a-9678-5dc7b3656bfe" (UID: "1016f833-117b-4e7a-9678-5dc7b3656bfe"). InnerVolumeSpecName "kube-api-access-5mqs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:23.223098 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.223086 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/325e2932-31f7-49a9-bac7-d8ac85a4500f-kube-api-access-jh9wt" pod "325e2932-31f7-49a9-bac7-d8ac85a4500f" (UID: "325e2932-31f7-49a9-bac7-d8ac85a4500f"). InnerVolumeSpecName "kube-api-access-jh9wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:23.322358 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.322323 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mqs9\" (UniqueName: \"kubernetes.io/projected/1016f833-117b-4e7a-9678-5dc7b3656bfe-kube-api-access-5mqs9\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:23.322358 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.322352 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh9wt\" (UniqueName: \"kubernetes.io/projected/325e2932-31f7-49a9-bac7-d8ac85a4500f-kube-api-access-jh9wt\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:23.409183 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.409096 4951 generic.go:356] "Generic (PLEG): container finished" podID="325e2932-31f7-49a9-bac7-d8ac85a4500f" containerID="d3a98c02056dade6e0fe7b9893c4c99d73d5a05b82ed88826f4304a4827590b4" exitCode=0 Oct 30 08:09:23.409183 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.409161 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4eff1f-76974db5db-jtmws" Oct 30 08:09:23.409183 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.409166 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4eff1f-76974db5db-jtmws" event={"ID":"325e2932-31f7-49a9-bac7-d8ac85a4500f","Type":"ContainerDied","Data":"d3a98c02056dade6e0fe7b9893c4c99d73d5a05b82ed88826f4304a4827590b4"} Oct 30 08:09:23.409477 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.409256 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4eff1f-76974db5db-jtmws" event={"ID":"325e2932-31f7-49a9-bac7-d8ac85a4500f","Type":"ContainerDied","Data":"c56d18a69b668639cf907a49f1b2a4d2871dfd734d598e2acebabcc09c9e00e2"} Oct 30 08:09:23.409477 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.409278 4951 scope.go:122] "RemoveContainer" containerID="d3a98c02056dade6e0fe7b9893c4c99d73d5a05b82ed88826f4304a4827590b4" Oct 30 08:09:23.410735 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.410713 4951 generic.go:356] "Generic (PLEG): container finished" podID="1016f833-117b-4e7a-9678-5dc7b3656bfe" containerID="5983b8f0bf5e6d7aec6cf088edd94c1ce059626ead3977cbe7dc7075e21457c3" exitCode=0 Oct 30 08:09:23.410873 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.410744 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4eff1f-76cb49c85f-vvqsc" event={"ID":"1016f833-117b-4e7a-9678-5dc7b3656bfe","Type":"ContainerDied","Data":"5983b8f0bf5e6d7aec6cf088edd94c1ce059626ead3977cbe7dc7075e21457c3"} Oct 30 08:09:23.410873 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.410773 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4eff1f-76cb49c85f-vvqsc" event={"ID":"1016f833-117b-4e7a-9678-5dc7b3656bfe","Type":"ContainerDied","Data":"d7776adccbefdefd0eab02ced76beab044596a850bf376b60f6243e1ed4d0e05"} Oct 30 08:09:23.410873 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.410781 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4eff1f-76cb49c85f-vvqsc" Oct 30 08:09:23.419997 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.419977 4951 scope.go:122] "RemoveContainer" containerID="d3a98c02056dade6e0fe7b9893c4c99d73d5a05b82ed88826f4304a4827590b4" Oct 30 08:09:23.420376 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:09:23.420352 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3a98c02056dade6e0fe7b9893c4c99d73d5a05b82ed88826f4304a4827590b4\": container with ID starting with d3a98c02056dade6e0fe7b9893c4c99d73d5a05b82ed88826f4304a4827590b4 not found: ID does not exist" containerID="d3a98c02056dade6e0fe7b9893c4c99d73d5a05b82ed88826f4304a4827590b4" Oct 30 08:09:23.420466 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.420390 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3a98c02056dade6e0fe7b9893c4c99d73d5a05b82ed88826f4304a4827590b4"} err="failed to get container status \"d3a98c02056dade6e0fe7b9893c4c99d73d5a05b82ed88826f4304a4827590b4\": rpc error: code = NotFound desc = could not find container \"d3a98c02056dade6e0fe7b9893c4c99d73d5a05b82ed88826f4304a4827590b4\": container with ID starting with d3a98c02056dade6e0fe7b9893c4c99d73d5a05b82ed88826f4304a4827590b4 not found: ID does not exist" Oct 30 08:09:23.420466 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.420417 4951 scope.go:122] "RemoveContainer" containerID="5983b8f0bf5e6d7aec6cf088edd94c1ce059626ead3977cbe7dc7075e21457c3" Oct 30 08:09:23.430978 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.430954 4951 scope.go:122] "RemoveContainer" containerID="5983b8f0bf5e6d7aec6cf088edd94c1ce059626ead3977cbe7dc7075e21457c3" Oct 30 08:09:23.431268 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:09:23.431246 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5983b8f0bf5e6d7aec6cf088edd94c1ce059626ead3977cbe7dc7075e21457c3\": container with ID starting with 5983b8f0bf5e6d7aec6cf088edd94c1ce059626ead3977cbe7dc7075e21457c3 not found: ID does not exist" containerID="5983b8f0bf5e6d7aec6cf088edd94c1ce059626ead3977cbe7dc7075e21457c3" Oct 30 08:09:23.431356 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.431279 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5983b8f0bf5e6d7aec6cf088edd94c1ce059626ead3977cbe7dc7075e21457c3"} err="failed to get container status \"5983b8f0bf5e6d7aec6cf088edd94c1ce059626ead3977cbe7dc7075e21457c3\": rpc error: code = NotFound desc = could not find container \"5983b8f0bf5e6d7aec6cf088edd94c1ce059626ead3977cbe7dc7075e21457c3\": container with ID starting with 5983b8f0bf5e6d7aec6cf088edd94c1ce059626ead3977cbe7dc7075e21457c3 not found: ID does not exist" Oct 30 08:09:23.463142 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.463101 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4eff1f-76974db5db-jtmws"] Oct 30 08:09:23.467461 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.467429 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4eff1f-76974db5db-jtmws"] Oct 30 08:09:23.500890 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.500856 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4eff1f-76cb49c85f-vvqsc"] Oct 30 08:09:23.507246 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.507216 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4eff1f-76cb49c85f-vvqsc"] Oct 30 08:09:23.769873 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.769841 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1016f833-117b-4e7a-9678-5dc7b3656bfe" path="/var/lib/kubelet/pods/1016f833-117b-4e7a-9678-5dc7b3656bfe/volumes" Oct 30 08:09:23.770289 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:23.770273 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="325e2932-31f7-49a9-bac7-d8ac85a4500f" path="/var/lib/kubelet/pods/325e2932-31f7-49a9-bac7-d8ac85a4500f/volumes" Oct 30 08:09:25.160771 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:25.160718 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-e5fa2d-6dd56c7bf6-96xb9" podUID="51ace3b8-b759-4dd3-bc27-be16dcdc9d97" containerName="payload-processing" containerID="cri-o://9a69722b7c209eb77333c093aaf7cde85f117f7329fafcd264d979fed50a4d71" gracePeriod=25 Oct 30 08:09:26.905154 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:26.905097 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:09:26.959727 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:26.959696 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a796d7df-446c-419e-be91-f479e9bfa8e2-istio-token\" (UniqueName: \"kubernetes.io/projected/a796d7df-446c-419e-be91-f479e9bfa8e2-istio-token\") pod \"a796d7df-446c-419e-be91-f479e9bfa8e2\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " Oct 30 08:09:26.959915 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:26.959740 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-workload-socket\") pod \"a796d7df-446c-419e-be91-f479e9bfa8e2\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " Oct 30 08:09:26.959915 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:26.959830 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-credential-socket\") pod \"a796d7df-446c-419e-be91-f479e9bfa8e2\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " Oct 30 08:09:26.959915 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:26.959864 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a796d7df-446c-419e-be91-f479e9bfa8e2-kube-api-access-dm7rp\" (UniqueName: \"kubernetes.io/projected/a796d7df-446c-419e-be91-f479e9bfa8e2-kube-api-access-dm7rp\") pod \"a796d7df-446c-419e-be91-f479e9bfa8e2\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " Oct 30 08:09:26.959915 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:26.959893 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-istio-data\") pod \"a796d7df-446c-419e-be91-f479e9bfa8e2\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " Oct 30 08:09:26.960152 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:26.959985 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-istio-envoy\") pod \"a796d7df-446c-419e-be91-f479e9bfa8e2\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " Oct 30 08:09:26.960152 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:26.960031 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a796d7df-446c-419e-be91-f479e9bfa8e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a796d7df-446c-419e-be91-f479e9bfa8e2-istio-podinfo\") pod \"a796d7df-446c-419e-be91-f479e9bfa8e2\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " Oct 30 08:09:26.960152 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:26.960066 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-workload-certs\") pod \"a796d7df-446c-419e-be91-f479e9bfa8e2\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " Oct 30 08:09:26.960152 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:26.960105 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a796d7df-446c-419e-be91-f479e9bfa8e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a796d7df-446c-419e-be91-f479e9bfa8e2-istiod-ca-cert\") pod \"a796d7df-446c-419e-be91-f479e9bfa8e2\" (UID: \"a796d7df-446c-419e-be91-f479e9bfa8e2\") " Oct 30 08:09:26.961345 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:26.960968 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-istio-data" pod "a796d7df-446c-419e-be91-f479e9bfa8e2" (UID: "a796d7df-446c-419e-be91-f479e9bfa8e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:26.961549 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:26.961503 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-credential-socket" pod "a796d7df-446c-419e-be91-f479e9bfa8e2" (UID: "a796d7df-446c-419e-be91-f479e9bfa8e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:26.961712 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:26.961690 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a796d7df-446c-419e-be91-f479e9bfa8e2-istiod-ca-cert" pod "a796d7df-446c-419e-be91-f479e9bfa8e2" (UID: "a796d7df-446c-419e-be91-f479e9bfa8e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:09:26.962022 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:26.961996 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-workload-certs" pod "a796d7df-446c-419e-be91-f479e9bfa8e2" (UID: "a796d7df-446c-419e-be91-f479e9bfa8e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:26.962022 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:26.962008 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-workload-socket" pod "a796d7df-446c-419e-be91-f479e9bfa8e2" (UID: "a796d7df-446c-419e-be91-f479e9bfa8e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:26.965084 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:26.965059 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-istio-envoy" pod "a796d7df-446c-419e-be91-f479e9bfa8e2" (UID: "a796d7df-446c-419e-be91-f479e9bfa8e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:26.970838 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:26.966154 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a796d7df-446c-419e-be91-f479e9bfa8e2-istio-token" pod "a796d7df-446c-419e-be91-f479e9bfa8e2" (UID: "a796d7df-446c-419e-be91-f479e9bfa8e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:26.980537 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:26.980510 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a796d7df-446c-419e-be91-f479e9bfa8e2-kube-api-access-dm7rp" pod "a796d7df-446c-419e-be91-f479e9bfa8e2" (UID: "a796d7df-446c-419e-be91-f479e9bfa8e2"). InnerVolumeSpecName "kube-api-access-dm7rp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:26.980751 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:26.980727 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a796d7df-446c-419e-be91-f479e9bfa8e2-istio-podinfo" pod "a796d7df-446c-419e-be91-f479e9bfa8e2" (UID: "a796d7df-446c-419e-be91-f479e9bfa8e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:09:27.061245 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:27.061215 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm7rp\" (UniqueName: \"kubernetes.io/projected/a796d7df-446c-419e-be91-f479e9bfa8e2-kube-api-access-dm7rp\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:27.061245 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:27.061245 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:27.061245 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:27.061258 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:27.061502 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:27.061274 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a796d7df-446c-419e-be91-f479e9bfa8e2-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:27.061502 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:27.061286 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:27.061502 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:27.061297 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a796d7df-446c-419e-be91-f479e9bfa8e2-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:27.061502 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:27.061309 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a796d7df-446c-419e-be91-f479e9bfa8e2-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:27.061502 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:27.061323 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:27.061502 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:27.061334 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a796d7df-446c-419e-be91-f479e9bfa8e2-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:27.436841 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:27.436802 4951 generic.go:356] "Generic (PLEG): container finished" podID="a796d7df-446c-419e-be91-f479e9bfa8e2" containerID="6bacce6fd9a94862334d7f540bf1296875d0445f548e36b69c7cab628ce7f2ef" exitCode=0 Oct 30 08:09:27.437080 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:27.436864 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" event={"ID":"a796d7df-446c-419e-be91-f479e9bfa8e2","Type":"ContainerDied","Data":"6bacce6fd9a94862334d7f540bf1296875d0445f548e36b69c7cab628ce7f2ef"} Oct 30 08:09:27.437080 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:27.436868 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" Oct 30 08:09:27.437080 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:27.436913 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j" event={"ID":"a796d7df-446c-419e-be91-f479e9bfa8e2","Type":"ContainerDied","Data":"5246d81fccd0df3b67b53ea055f312a6bdb49629de5f317de58ff6784c9b9008"} Oct 30 08:09:27.437080 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:27.436945 4951 scope.go:122] "RemoveContainer" containerID="6bacce6fd9a94862334d7f540bf1296875d0445f548e36b69c7cab628ce7f2ef" Oct 30 08:09:27.447013 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:27.446988 4951 scope.go:122] "RemoveContainer" containerID="6bacce6fd9a94862334d7f540bf1296875d0445f548e36b69c7cab628ce7f2ef" Oct 30 08:09:27.447452 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:09:27.447417 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bacce6fd9a94862334d7f540bf1296875d0445f548e36b69c7cab628ce7f2ef\": container with ID starting with 6bacce6fd9a94862334d7f540bf1296875d0445f548e36b69c7cab628ce7f2ef not found: ID does not exist" containerID="6bacce6fd9a94862334d7f540bf1296875d0445f548e36b69c7cab628ce7f2ef" Oct 30 08:09:27.447648 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:27.447461 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bacce6fd9a94862334d7f540bf1296875d0445f548e36b69c7cab628ce7f2ef"} err="failed to get container status \"6bacce6fd9a94862334d7f540bf1296875d0445f548e36b69c7cab628ce7f2ef\": rpc error: code = NotFound desc = could not find container \"6bacce6fd9a94862334d7f540bf1296875d0445f548e36b69c7cab628ce7f2ef\": container with ID starting with 6bacce6fd9a94862334d7f540bf1296875d0445f548e36b69c7cab628ce7f2ef not found: ID does not exist" Oct 30 08:09:27.491627 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:27.491587 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j"] Oct 30 08:09:27.496959 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:27.496928 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-vps9j"] Oct 30 08:09:27.770957 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:27.770921 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a796d7df-446c-419e-be91-f479e9bfa8e2" path="/var/lib/kubelet/pods/a796d7df-446c-419e-be91-f479e9bfa8e2/volumes" Oct 30 08:09:28.400450 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:28.400409 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-e5fa2d-6dd56c7bf6-96xb9" podUID="51ace3b8-b759-4dd3-bc27-be16dcdc9d97" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:09:29.430880 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:29.430834 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" containerID="cri-o://c3076f46ec1ee0747238205ec259f639539225b459d68004da03f1a89ff3c791" gracePeriod=45 Oct 30 08:09:29.690136 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:29.690062 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 08:09:29.798620 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:29.798560 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb929374-4eb9-4a81-b81a-b89098b9fb20-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb929374-4eb9-4a81-b81a-b89098b9fb20-tmp-dir\") pod \"bb929374-4eb9-4a81-b81a-b89098b9fb20\" (UID: \"bb929374-4eb9-4a81-b81a-b89098b9fb20\") " Oct 30 08:09:29.798841 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:29.798651 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb929374-4eb9-4a81-b81a-b89098b9fb20-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb929374-4eb9-4a81-b81a-b89098b9fb20-dshm\") pod \"bb929374-4eb9-4a81-b81a-b89098b9fb20\" (UID: \"bb929374-4eb9-4a81-b81a-b89098b9fb20\") " Oct 30 08:09:29.798841 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:29.798691 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb929374-4eb9-4a81-b81a-b89098b9fb20-kube-api-access-sc5hx\" (UniqueName: \"kubernetes.io/projected/bb929374-4eb9-4a81-b81a-b89098b9fb20-kube-api-access-sc5hx\") pod \"bb929374-4eb9-4a81-b81a-b89098b9fb20\" (UID: \"bb929374-4eb9-4a81-b81a-b89098b9fb20\") " Oct 30 08:09:29.798841 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:29.798723 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb929374-4eb9-4a81-b81a-b89098b9fb20-home\" (UniqueName: \"kubernetes.io/empty-dir/bb929374-4eb9-4a81-b81a-b89098b9fb20-home\") pod \"bb929374-4eb9-4a81-b81a-b89098b9fb20\" (UID: \"bb929374-4eb9-4a81-b81a-b89098b9fb20\") " Oct 30 08:09:29.799088 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:29.798849 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb929374-4eb9-4a81-b81a-b89098b9fb20-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb929374-4eb9-4a81-b81a-b89098b9fb20-model-cache\") pod \"bb929374-4eb9-4a81-b81a-b89098b9fb20\" (UID: \"bb929374-4eb9-4a81-b81a-b89098b9fb20\") " Oct 30 08:09:29.799088 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:29.798882 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb929374-4eb9-4a81-b81a-b89098b9fb20-tmp-dir" pod "bb929374-4eb9-4a81-b81a-b89098b9fb20" (UID: "bb929374-4eb9-4a81-b81a-b89098b9fb20"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:29.799088 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:29.798961 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb929374-4eb9-4a81-b81a-b89098b9fb20-tls-certs\" (UniqueName: \"kubernetes.io/secret/bb929374-4eb9-4a81-b81a-b89098b9fb20-tls-certs\") pod \"bb929374-4eb9-4a81-b81a-b89098b9fb20\" (UID: \"bb929374-4eb9-4a81-b81a-b89098b9fb20\") " Oct 30 08:09:29.799088 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:29.799051 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb929374-4eb9-4a81-b81a-b89098b9fb20-home" pod "bb929374-4eb9-4a81-b81a-b89098b9fb20" (UID: "bb929374-4eb9-4a81-b81a-b89098b9fb20"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:29.799340 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:29.799255 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb929374-4eb9-4a81-b81a-b89098b9fb20-model-cache" pod "bb929374-4eb9-4a81-b81a-b89098b9fb20" (UID: "bb929374-4eb9-4a81-b81a-b89098b9fb20"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:29.799423 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:29.799399 4951 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb929374-4eb9-4a81-b81a-b89098b9fb20-tmp-dir\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:29.799423 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:29.799422 4951 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb929374-4eb9-4a81-b81a-b89098b9fb20-home\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:29.799423 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:29.799435 4951 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb929374-4eb9-4a81-b81a-b89098b9fb20-model-cache\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:29.801395 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:29.801367 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb929374-4eb9-4a81-b81a-b89098b9fb20-tls-certs" pod "bb929374-4eb9-4a81-b81a-b89098b9fb20" (UID: "bb929374-4eb9-4a81-b81a-b89098b9fb20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:09:29.801395 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:29.801378 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb929374-4eb9-4a81-b81a-b89098b9fb20-kube-api-access-sc5hx" pod "bb929374-4eb9-4a81-b81a-b89098b9fb20" (UID: "bb929374-4eb9-4a81-b81a-b89098b9fb20"). InnerVolumeSpecName "kube-api-access-sc5hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:29.801744 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:29.801727 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb929374-4eb9-4a81-b81a-b89098b9fb20-dshm" pod "bb929374-4eb9-4a81-b81a-b89098b9fb20" (UID: "bb929374-4eb9-4a81-b81a-b89098b9fb20"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:29.900339 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:29.900287 4951 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb929374-4eb9-4a81-b81a-b89098b9fb20-dshm\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:29.900339 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:29.900333 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc5hx\" (UniqueName: \"kubernetes.io/projected/bb929374-4eb9-4a81-b81a-b89098b9fb20-kube-api-access-sc5hx\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:29.900339 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:29.900349 4951 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb929374-4eb9-4a81-b81a-b89098b9fb20-tls-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:30.453416 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:30.453375 4951 generic.go:356] "Generic (PLEG): container finished" podID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerID="c3076f46ec1ee0747238205ec259f639539225b459d68004da03f1a89ff3c791" exitCode=0 Oct 30 08:09:30.453905 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:30.453415 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" event={"ID":"bb929374-4eb9-4a81-b81a-b89098b9fb20","Type":"ContainerDied","Data":"c3076f46ec1ee0747238205ec259f639539225b459d68004da03f1a89ff3c791"} Oct 30 08:09:30.453905 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:30.453463 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" event={"ID":"bb929374-4eb9-4a81-b81a-b89098b9fb20","Type":"ContainerDied","Data":"6b656b7702b589391d33edd3778363fd2c04b0d4749a4992346c94a75ad5c0da"} Oct 30 08:09:30.453905 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:30.453480 4951 scope.go:122] "RemoveContainer" containerID="c3076f46ec1ee0747238205ec259f639539225b459d68004da03f1a89ff3c791" Oct 30 08:09:30.453905 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:30.453421 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792" Oct 30 08:09:30.468310 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:30.465882 4951 scope.go:122] "RemoveContainer" containerID="c3076f46ec1ee0747238205ec259f639539225b459d68004da03f1a89ff3c791" Oct 30 08:09:30.468907 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:09:30.468477 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3076f46ec1ee0747238205ec259f639539225b459d68004da03f1a89ff3c791\": container with ID starting with c3076f46ec1ee0747238205ec259f639539225b459d68004da03f1a89ff3c791 not found: ID does not exist" containerID="c3076f46ec1ee0747238205ec259f639539225b459d68004da03f1a89ff3c791" Oct 30 08:09:30.468907 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:30.468518 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3076f46ec1ee0747238205ec259f639539225b459d68004da03f1a89ff3c791"} err="failed to get container status \"c3076f46ec1ee0747238205ec259f639539225b459d68004da03f1a89ff3c791\": rpc error: code = NotFound desc = could not find container \"c3076f46ec1ee0747238205ec259f639539225b459d68004da03f1a89ff3c791\": container with ID starting with c3076f46ec1ee0747238205ec259f639539225b459d68004da03f1a89ff3c791 not found: ID does not exist" Oct 30 08:09:30.507891 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:30.507860 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792"] Oct 30 08:09:30.519104 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:30.519076 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e5fa2d/distinct-w2-e5fa2d-kserve-b68f59c4b-df792"] Oct 30 08:09:31.770203 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:31.770170 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" path="/var/lib/kubelet/pods/bb929374-4eb9-4a81-b81a-b89098b9fb20/volumes" Oct 30 08:09:32.830777 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:32.830745 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2f845b-7d977fd55d-5ctj5"] Oct 30 08:09:32.866329 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:32.866302 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-2f845b-5d5d94cf95-jsspt"] Oct 30 08:09:37.831676 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:37.831628 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-2f845b-7d977fd55d-5ctj5" podUID="798d3036-be81-460a-819d-2385a159ecb1" containerName="payload-processing" containerID="cri-o://ef8f638005f7a4c7849ac2edd2c19c73b9bb584d829cef3a9547fbadba632bbe" gracePeriod=25 Oct 30 08:09:37.867384 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:37.867338 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2f845b-5d5d94cf95-jsspt" podUID="4b05487e-0050-4d5a-8875-4878d2b565a4" containerName="payload-pre-processing" containerID="cri-o://30dc748d281ff7f18235b353bc665df4684dc71ffbd616a6590039b84470d454" gracePeriod=25 Oct 30 08:09:38.400041 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:38.399991 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-e5fa2d-6dd56c7bf6-96xb9" podUID="51ace3b8-b759-4dd3-bc27-be16dcdc9d97" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:09:45.366130 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:45.366093 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2f845b-5d5d94cf95-jsspt" podUID="4b05487e-0050-4d5a-8875-4878d2b565a4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:09:45.366703 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:45.366473 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-2f845b-7d977fd55d-5ctj5" podUID="798d3036-be81-460a-819d-2385a159ecb1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:09:45.420744 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:45.420691 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e5fa2d-6dd56c7bf6-96xb9" Oct 30 08:09:45.540962 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:45.540919 4951 generic.go:356] "Generic (PLEG): container finished" podID="51ace3b8-b759-4dd3-bc27-be16dcdc9d97" containerID="9a69722b7c209eb77333c093aaf7cde85f117f7329fafcd264d979fed50a4d71" exitCode=0 Oct 30 08:09:45.540962 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:45.540961 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e5fa2d-6dd56c7bf6-96xb9" Oct 30 08:09:45.541253 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:45.540995 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e5fa2d-6dd56c7bf6-96xb9" event={"ID":"51ace3b8-b759-4dd3-bc27-be16dcdc9d97","Type":"ContainerDied","Data":"9a69722b7c209eb77333c093aaf7cde85f117f7329fafcd264d979fed50a4d71"} Oct 30 08:09:45.541253 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:45.541058 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e5fa2d-6dd56c7bf6-96xb9" event={"ID":"51ace3b8-b759-4dd3-bc27-be16dcdc9d97","Type":"ContainerDied","Data":"79be54ec934a7a5f9a907ad4a362344132e8fb7355ccc376e59e824b69402703"} Oct 30 08:09:45.541253 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:45.541082 4951 scope.go:122] "RemoveContainer" containerID="9a69722b7c209eb77333c093aaf7cde85f117f7329fafcd264d979fed50a4d71" Oct 30 08:09:45.549831 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:45.549805 4951 scope.go:122] "RemoveContainer" containerID="9a69722b7c209eb77333c093aaf7cde85f117f7329fafcd264d979fed50a4d71" Oct 30 08:09:45.550113 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:09:45.550094 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a69722b7c209eb77333c093aaf7cde85f117f7329fafcd264d979fed50a4d71\": container with ID starting with 9a69722b7c209eb77333c093aaf7cde85f117f7329fafcd264d979fed50a4d71 not found: ID does not exist" containerID="9a69722b7c209eb77333c093aaf7cde85f117f7329fafcd264d979fed50a4d71" Oct 30 08:09:45.550197 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:45.550124 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a69722b7c209eb77333c093aaf7cde85f117f7329fafcd264d979fed50a4d71"} err="failed to get container status \"9a69722b7c209eb77333c093aaf7cde85f117f7329fafcd264d979fed50a4d71\": rpc error: code = NotFound desc = could not find container \"9a69722b7c209eb77333c093aaf7cde85f117f7329fafcd264d979fed50a4d71\": container with ID starting with 9a69722b7c209eb77333c093aaf7cde85f117f7329fafcd264d979fed50a4d71 not found: ID does not exist" Oct 30 08:09:45.554547 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:45.554518 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51ace3b8-b759-4dd3-bc27-be16dcdc9d97-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51ace3b8-b759-4dd3-bc27-be16dcdc9d97-plugins-config-volume\") pod \"51ace3b8-b759-4dd3-bc27-be16dcdc9d97\" (UID: \"51ace3b8-b759-4dd3-bc27-be16dcdc9d97\") " Oct 30 08:09:45.554727 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:45.554552 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51ace3b8-b759-4dd3-bc27-be16dcdc9d97-kube-api-access-7d756\" (UniqueName: \"kubernetes.io/projected/51ace3b8-b759-4dd3-bc27-be16dcdc9d97-kube-api-access-7d756\") pod \"51ace3b8-b759-4dd3-bc27-be16dcdc9d97\" (UID: \"51ace3b8-b759-4dd3-bc27-be16dcdc9d97\") " Oct 30 08:09:45.554877 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:45.554856 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51ace3b8-b759-4dd3-bc27-be16dcdc9d97-plugins-config-volume" pod "51ace3b8-b759-4dd3-bc27-be16dcdc9d97" (UID: "51ace3b8-b759-4dd3-bc27-be16dcdc9d97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:09:45.556827 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:45.556802 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51ace3b8-b759-4dd3-bc27-be16dcdc9d97-kube-api-access-7d756" pod "51ace3b8-b759-4dd3-bc27-be16dcdc9d97" (UID: "51ace3b8-b759-4dd3-bc27-be16dcdc9d97"). InnerVolumeSpecName "kube-api-access-7d756". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:45.655626 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:45.655593 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51ace3b8-b759-4dd3-bc27-be16dcdc9d97-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:45.655626 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:45.655623 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d756\" (UniqueName: \"kubernetes.io/projected/51ace3b8-b759-4dd3-bc27-be16dcdc9d97-kube-api-access-7d756\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:45.882307 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:45.882274 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e5fa2d-6dd56c7bf6-96xb9"] Oct 30 08:09:45.894923 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:45.894885 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e5fa2d-6dd56c7bf6-96xb9"] Oct 30 08:09:45.943866 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:45.943834 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf"] Oct 30 08:09:45.944114 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:45.944083 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf" podUID="dc75560d-af65-4bad-9bd4-c7e7156882be" containerName="maas-api" containerID="cri-o://50257b1188bff7f3033d259b209a8f5057134d5378fb5582cf9e1fc287c1c053" gracePeriod=30 Oct 30 08:09:46.206258 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:46.206231 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf" Oct 30 08:09:46.364312 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:46.364205 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc75560d-af65-4bad-9bd4-c7e7156882be-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc75560d-af65-4bad-9bd4-c7e7156882be-metrics-certs\") pod \"dc75560d-af65-4bad-9bd4-c7e7156882be\" (UID: \"dc75560d-af65-4bad-9bd4-c7e7156882be\") " Oct 30 08:09:46.364594 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:46.364354 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc75560d-af65-4bad-9bd4-c7e7156882be-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc75560d-af65-4bad-9bd4-c7e7156882be-openshift-service-ca\") pod \"dc75560d-af65-4bad-9bd4-c7e7156882be\" (UID: \"dc75560d-af65-4bad-9bd4-c7e7156882be\") " Oct 30 08:09:46.364594 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:46.364417 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc75560d-af65-4bad-9bd4-c7e7156882be-kube-api-access-hhwpg\" (UniqueName: \"kubernetes.io/projected/dc75560d-af65-4bad-9bd4-c7e7156882be-kube-api-access-hhwpg\") pod \"dc75560d-af65-4bad-9bd4-c7e7156882be\" (UID: \"dc75560d-af65-4bad-9bd4-c7e7156882be\") " Oct 30 08:09:46.364594 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:46.364462 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc75560d-af65-4bad-9bd4-c7e7156882be-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc75560d-af65-4bad-9bd4-c7e7156882be-maas-api-tls\") pod \"dc75560d-af65-4bad-9bd4-c7e7156882be\" (UID: \"dc75560d-af65-4bad-9bd4-c7e7156882be\") " Oct 30 08:09:46.365799 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:46.365762 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc75560d-af65-4bad-9bd4-c7e7156882be-openshift-service-ca" pod "dc75560d-af65-4bad-9bd4-c7e7156882be" (UID: "dc75560d-af65-4bad-9bd4-c7e7156882be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:09:46.367834 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:46.367805 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc75560d-af65-4bad-9bd4-c7e7156882be-maas-api-tls" pod "dc75560d-af65-4bad-9bd4-c7e7156882be" (UID: "dc75560d-af65-4bad-9bd4-c7e7156882be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:09:46.368211 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:46.368187 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc75560d-af65-4bad-9bd4-c7e7156882be-kube-api-access-hhwpg" pod "dc75560d-af65-4bad-9bd4-c7e7156882be" (UID: "dc75560d-af65-4bad-9bd4-c7e7156882be"). InnerVolumeSpecName "kube-api-access-hhwpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:46.373375 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:46.373351 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc75560d-af65-4bad-9bd4-c7e7156882be-metrics-certs" pod "dc75560d-af65-4bad-9bd4-c7e7156882be" (UID: "dc75560d-af65-4bad-9bd4-c7e7156882be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:09:46.465950 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:46.465897 4951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc75560d-af65-4bad-9bd4-c7e7156882be-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:46.465950 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:46.465940 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhwpg\" (UniqueName: \"kubernetes.io/projected/dc75560d-af65-4bad-9bd4-c7e7156882be-kube-api-access-hhwpg\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:46.465950 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:46.465956 4951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc75560d-af65-4bad-9bd4-c7e7156882be-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:46.466296 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:46.465970 4951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc75560d-af65-4bad-9bd4-c7e7156882be-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:46.549231 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:46.549192 4951 generic.go:356] "Generic (PLEG): container finished" podID="dc75560d-af65-4bad-9bd4-c7e7156882be" containerID="50257b1188bff7f3033d259b209a8f5057134d5378fb5582cf9e1fc287c1c053" exitCode=0 Oct 30 08:09:46.549434 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:46.549233 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf" Oct 30 08:09:46.549434 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:46.549271 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf" event={"ID":"dc75560d-af65-4bad-9bd4-c7e7156882be","Type":"ContainerDied","Data":"50257b1188bff7f3033d259b209a8f5057134d5378fb5582cf9e1fc287c1c053"} Oct 30 08:09:46.549434 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:46.549331 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf" event={"ID":"dc75560d-af65-4bad-9bd4-c7e7156882be","Type":"ContainerDied","Data":"32d1365b704d6f0b4bc6065b171d3e90c23a7372d29819a988d06797e5ec210d"} Oct 30 08:09:46.549434 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:46.549361 4951 scope.go:122] "RemoveContainer" containerID="50257b1188bff7f3033d259b209a8f5057134d5378fb5582cf9e1fc287c1c053" Oct 30 08:09:46.558902 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:46.558886 4951 scope.go:122] "RemoveContainer" containerID="50257b1188bff7f3033d259b209a8f5057134d5378fb5582cf9e1fc287c1c053" Oct 30 08:09:46.559165 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:09:46.559142 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50257b1188bff7f3033d259b209a8f5057134d5378fb5582cf9e1fc287c1c053\": container with ID starting with 50257b1188bff7f3033d259b209a8f5057134d5378fb5582cf9e1fc287c1c053 not found: ID does not exist" containerID="50257b1188bff7f3033d259b209a8f5057134d5378fb5582cf9e1fc287c1c053" Oct 30 08:09:46.559237 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:46.559173 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50257b1188bff7f3033d259b209a8f5057134d5378fb5582cf9e1fc287c1c053"} err="failed to get container status \"50257b1188bff7f3033d259b209a8f5057134d5378fb5582cf9e1fc287c1c053\": rpc error: code = NotFound desc = could not find container \"50257b1188bff7f3033d259b209a8f5057134d5378fb5582cf9e1fc287c1c053\": container with ID starting with 50257b1188bff7f3033d259b209a8f5057134d5378fb5582cf9e1fc287c1c053 not found: ID does not exist" Oct 30 08:09:46.601533 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:46.601505 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf"] Oct 30 08:09:46.614065 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:46.614042 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2f845b-699b4bbf69-vbczf"] Oct 30 08:09:47.771028 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:47.770991 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51ace3b8-b759-4dd3-bc27-be16dcdc9d97" path="/var/lib/kubelet/pods/51ace3b8-b759-4dd3-bc27-be16dcdc9d97/volumes" Oct 30 08:09:47.771568 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:47.771548 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc75560d-af65-4bad-9bd4-c7e7156882be" path="/var/lib/kubelet/pods/dc75560d-af65-4bad-9bd4-c7e7156882be/volumes" Oct 30 08:09:53.021889 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:09:53.021853 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 08:09:54.648488 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:54.648455 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc"] Oct 30 08:09:54.649079 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:54.648680 4951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" podUID="f22099c5-3928-4f09-b587-55bedc15e727" containerName="istio-proxy" containerID="cri-o://71179fdc9d969c06970e3b5bc6f4517d98daacb5570c0bdd3ce9d68a84f05de2" gracePeriod=30 Oct 30 08:09:55.366773 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:55.366730 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2f845b-5d5d94cf95-jsspt" podUID="4b05487e-0050-4d5a-8875-4878d2b565a4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:09:55.367039 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:55.366807 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-2f845b-7d977fd55d-5ctj5" podUID="798d3036-be81-460a-819d-2385a159ecb1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:09:55.367039 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:55.366860 4951 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 08:09:55.367039 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:55.366902 4951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" podUID="f22099c5-3928-4f09-b587-55bedc15e727" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 08:09:56.883747 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:56.883709 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-nk2g9_454b78a6-5a90-41a9-8ff5-fa16edf2b257/manager/0.log" Oct 30 08:09:58.095814 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:58.095786 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-795446ff4c-nbl2l_483fcce9-07d5-4224-bfc9-6ae15c087276/manager/0.log" Oct 30 08:09:58.145827 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:58.145803 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-2f845b-7d977fd55d-5ctj5" Oct 30 08:09:58.170078 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:58.170053 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2f845b-5d5d94cf95-jsspt" Oct 30 08:09:58.183682 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:58.183659 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/798d3036-be81-460a-819d-2385a159ecb1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/798d3036-be81-460a-819d-2385a159ecb1-plugins-config-volume\") pod \"798d3036-be81-460a-819d-2385a159ecb1\" (UID: \"798d3036-be81-460a-819d-2385a159ecb1\") " Oct 30 08:09:58.183786 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:58.183722 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b05487e-0050-4d5a-8875-4878d2b565a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b05487e-0050-4d5a-8875-4878d2b565a4-plugins-config-volume\") pod \"4b05487e-0050-4d5a-8875-4878d2b565a4\" (UID: \"4b05487e-0050-4d5a-8875-4878d2b565a4\") " Oct 30 08:09:58.183786 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:58.183772 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/798d3036-be81-460a-819d-2385a159ecb1-kube-api-access-wfhkn\" (UniqueName: \"kubernetes.io/projected/798d3036-be81-460a-819d-2385a159ecb1-kube-api-access-wfhkn\") pod \"798d3036-be81-460a-819d-2385a159ecb1\" (UID: \"798d3036-be81-460a-819d-2385a159ecb1\") " Oct 30 08:09:58.183881 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:58.183804 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b05487e-0050-4d5a-8875-4878d2b565a4-kube-api-access-zp6vt\" (UniqueName: \"kubernetes.io/projected/4b05487e-0050-4d5a-8875-4878d2b565a4-kube-api-access-zp6vt\") pod \"4b05487e-0050-4d5a-8875-4878d2b565a4\" (UID: \"4b05487e-0050-4d5a-8875-4878d2b565a4\") " Oct 30 08:09:58.184120 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:58.184092 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/798d3036-be81-460a-819d-2385a159ecb1-plugins-config-volume" pod "798d3036-be81-460a-819d-2385a159ecb1" (UID: "798d3036-be81-460a-819d-2385a159ecb1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:09:58.184220 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:58.184117 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b05487e-0050-4d5a-8875-4878d2b565a4-plugins-config-volume" pod "4b05487e-0050-4d5a-8875-4878d2b565a4" (UID: "4b05487e-0050-4d5a-8875-4878d2b565a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:09:58.185789 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:58.185768 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/798d3036-be81-460a-819d-2385a159ecb1-kube-api-access-wfhkn" pod "798d3036-be81-460a-819d-2385a159ecb1" (UID: "798d3036-be81-460a-819d-2385a159ecb1"). InnerVolumeSpecName "kube-api-access-wfhkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:58.185873 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:58.185808 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b05487e-0050-4d5a-8875-4878d2b565a4-kube-api-access-zp6vt" pod "4b05487e-0050-4d5a-8875-4878d2b565a4" (UID: "4b05487e-0050-4d5a-8875-4878d2b565a4"). InnerVolumeSpecName "kube-api-access-zp6vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:58.284502 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:58.284465 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfhkn\" (UniqueName: \"kubernetes.io/projected/798d3036-be81-460a-819d-2385a159ecb1-kube-api-access-wfhkn\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:58.284502 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:58.284491 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp6vt\" (UniqueName: \"kubernetes.io/projected/4b05487e-0050-4d5a-8875-4878d2b565a4-kube-api-access-zp6vt\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:58.284502 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:58.284501 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/798d3036-be81-460a-819d-2385a159ecb1-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:58.284502 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:58.284511 4951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b05487e-0050-4d5a-8875-4878d2b565a4-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:09:58.607597 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:58.607536 4951 generic.go:356] "Generic (PLEG): container finished" podID="798d3036-be81-460a-819d-2385a159ecb1" containerID="ef8f638005f7a4c7849ac2edd2c19c73b9bb584d829cef3a9547fbadba632bbe" exitCode=0 Oct 30 08:09:58.607802 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:58.607602 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-2f845b-7d977fd55d-5ctj5" Oct 30 08:09:58.607802 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:58.607626 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-2f845b-7d977fd55d-5ctj5" event={"ID":"798d3036-be81-460a-819d-2385a159ecb1","Type":"ContainerDied","Data":"ef8f638005f7a4c7849ac2edd2c19c73b9bb584d829cef3a9547fbadba632bbe"} Oct 30 08:09:58.607802 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:58.607681 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-2f845b-7d977fd55d-5ctj5" event={"ID":"798d3036-be81-460a-819d-2385a159ecb1","Type":"ContainerDied","Data":"5a3b94bd12be78c4b315284e1cbccd72eba5d89fc85c615b92d7cd07b1b33db2"} Oct 30 08:09:58.607802 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:58.607706 4951 scope.go:122] "RemoveContainer" containerID="ef8f638005f7a4c7849ac2edd2c19c73b9bb584d829cef3a9547fbadba632bbe" Oct 30 08:09:58.609255 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:58.609231 4951 generic.go:356] "Generic (PLEG): container finished" podID="4b05487e-0050-4d5a-8875-4878d2b565a4" containerID="30dc748d281ff7f18235b353bc665df4684dc71ffbd616a6590039b84470d454" exitCode=0 Oct 30 08:09:58.609386 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:58.609285 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2f845b-5d5d94cf95-jsspt" Oct 30 08:09:58.609500 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:58.609469 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2f845b-5d5d94cf95-jsspt" event={"ID":"4b05487e-0050-4d5a-8875-4878d2b565a4","Type":"ContainerDied","Data":"30dc748d281ff7f18235b353bc665df4684dc71ffbd616a6590039b84470d454"} Oct 30 08:09:58.609646 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:58.609524 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2f845b-5d5d94cf95-jsspt" event={"ID":"4b05487e-0050-4d5a-8875-4878d2b565a4","Type":"ContainerDied","Data":"90f2ce31b9400c98951c64bc3b8b8e11497c7b79ee1ab99e72ce6a7cabb2247d"} Oct 30 08:09:58.617555 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:58.617537 4951 scope.go:122] "RemoveContainer" containerID="ef8f638005f7a4c7849ac2edd2c19c73b9bb584d829cef3a9547fbadba632bbe" Oct 30 08:09:58.617848 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:09:58.617828 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef8f638005f7a4c7849ac2edd2c19c73b9bb584d829cef3a9547fbadba632bbe\": container with ID starting with ef8f638005f7a4c7849ac2edd2c19c73b9bb584d829cef3a9547fbadba632bbe not found: ID does not exist" containerID="ef8f638005f7a4c7849ac2edd2c19c73b9bb584d829cef3a9547fbadba632bbe" Oct 30 08:09:58.617911 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:58.617856 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef8f638005f7a4c7849ac2edd2c19c73b9bb584d829cef3a9547fbadba632bbe"} err="failed to get container status \"ef8f638005f7a4c7849ac2edd2c19c73b9bb584d829cef3a9547fbadba632bbe\": rpc error: code = NotFound desc = could not find container \"ef8f638005f7a4c7849ac2edd2c19c73b9bb584d829cef3a9547fbadba632bbe\": container with ID starting with ef8f638005f7a4c7849ac2edd2c19c73b9bb584d829cef3a9547fbadba632bbe not found: ID does not exist" Oct 30 08:09:58.617911 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:58.617872 4951 scope.go:122] "RemoveContainer" containerID="30dc748d281ff7f18235b353bc665df4684dc71ffbd616a6590039b84470d454" Oct 30 08:09:58.625731 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:58.625716 4951 scope.go:122] "RemoveContainer" containerID="30dc748d281ff7f18235b353bc665df4684dc71ffbd616a6590039b84470d454" Oct 30 08:09:58.625984 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:09:58.625966 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30dc748d281ff7f18235b353bc665df4684dc71ffbd616a6590039b84470d454\": container with ID starting with 30dc748d281ff7f18235b353bc665df4684dc71ffbd616a6590039b84470d454 not found: ID does not exist" containerID="30dc748d281ff7f18235b353bc665df4684dc71ffbd616a6590039b84470d454" Oct 30 08:09:58.626032 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:58.625991 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30dc748d281ff7f18235b353bc665df4684dc71ffbd616a6590039b84470d454"} err="failed to get container status \"30dc748d281ff7f18235b353bc665df4684dc71ffbd616a6590039b84470d454\": rpc error: code = NotFound desc = could not find container \"30dc748d281ff7f18235b353bc665df4684dc71ffbd616a6590039b84470d454\": container with ID starting with 30dc748d281ff7f18235b353bc665df4684dc71ffbd616a6590039b84470d454 not found: ID does not exist" Oct 30 08:09:58.660620 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:58.660600 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-2f845b-5d5d94cf95-jsspt"] Oct 30 08:09:58.665828 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:58.665811 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-2f845b-5d5d94cf95-jsspt"] Oct 30 08:09:58.701803 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:58.701777 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2f845b-7d977fd55d-5ctj5"] Oct 30 08:09:58.710336 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:58.710308 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2f845b-7d977fd55d-5ctj5"] Oct 30 08:09:59.770760 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:59.770725 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b05487e-0050-4d5a-8875-4878d2b565a4" path="/var/lib/kubelet/pods/4b05487e-0050-4d5a-8875-4878d2b565a4/volumes" Oct 30 08:09:59.771137 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:59.771122 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="798d3036-be81-460a-819d-2385a159ecb1" path="/var/lib/kubelet/pods/798d3036-be81-460a-819d-2385a159ecb1/volumes" Oct 30 08:09:59.904508 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:09:59.904479 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 08:10:00.000367 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.000342 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f22099c5-3928-4f09-b587-55bedc15e727-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f22099c5-3928-4f09-b587-55bedc15e727-istiod-ca-cert\") pod \"f22099c5-3928-4f09-b587-55bedc15e727\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " Oct 30 08:10:00.000512 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.000394 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f22099c5-3928-4f09-b587-55bedc15e727-kube-api-access-vghz8\" (UniqueName: \"kubernetes.io/projected/f22099c5-3928-4f09-b587-55bedc15e727-kube-api-access-vghz8\") pod \"f22099c5-3928-4f09-b587-55bedc15e727\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " Oct 30 08:10:00.000512 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.000422 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-credential-socket\") pod \"f22099c5-3928-4f09-b587-55bedc15e727\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " Oct 30 08:10:00.000512 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.000437 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f22099c5-3928-4f09-b587-55bedc15e727-istio-token\" (UniqueName: \"kubernetes.io/projected/f22099c5-3928-4f09-b587-55bedc15e727-istio-token\") pod \"f22099c5-3928-4f09-b587-55bedc15e727\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " Oct 30 08:10:00.000512 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.000454 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-workload-certs\") pod \"f22099c5-3928-4f09-b587-55bedc15e727\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " Oct 30 08:10:00.000512 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.000489 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-workload-socket\") pod \"f22099c5-3928-4f09-b587-55bedc15e727\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " Oct 30 08:10:00.000780 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.000537 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-istio-envoy\") pod \"f22099c5-3928-4f09-b587-55bedc15e727\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " Oct 30 08:10:00.000780 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.000563 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f22099c5-3928-4f09-b587-55bedc15e727-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f22099c5-3928-4f09-b587-55bedc15e727-istio-podinfo\") pod \"f22099c5-3928-4f09-b587-55bedc15e727\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " Oct 30 08:10:00.000780 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.000627 4951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-istio-data\") pod \"f22099c5-3928-4f09-b587-55bedc15e727\" (UID: \"f22099c5-3928-4f09-b587-55bedc15e727\") " Oct 30 08:10:00.000950 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.000810 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-credential-socket" pod "f22099c5-3928-4f09-b587-55bedc15e727" (UID: "f22099c5-3928-4f09-b587-55bedc15e727"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:10:00.000950 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.000847 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f22099c5-3928-4f09-b587-55bedc15e727-istiod-ca-cert" pod "f22099c5-3928-4f09-b587-55bedc15e727" (UID: "f22099c5-3928-4f09-b587-55bedc15e727"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:10:00.001074 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.000982 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-workload-certs" pod "f22099c5-3928-4f09-b587-55bedc15e727" (UID: "f22099c5-3928-4f09-b587-55bedc15e727"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:10:00.001074 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.001019 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-istio-data" pod "f22099c5-3928-4f09-b587-55bedc15e727" (UID: "f22099c5-3928-4f09-b587-55bedc15e727"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:10:00.001074 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.001043 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-workload-socket" pod "f22099c5-3928-4f09-b587-55bedc15e727" (UID: "f22099c5-3928-4f09-b587-55bedc15e727"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:10:00.001074 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.001063 4951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:10:00.001224 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.001083 4951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f22099c5-3928-4f09-b587-55bedc15e727-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:10:00.002727 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.002700 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f22099c5-3928-4f09-b587-55bedc15e727-kube-api-access-vghz8" pod "f22099c5-3928-4f09-b587-55bedc15e727" (UID: "f22099c5-3928-4f09-b587-55bedc15e727"). InnerVolumeSpecName "kube-api-access-vghz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:10:00.002887 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.002869 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f22099c5-3928-4f09-b587-55bedc15e727-istio-podinfo" pod "f22099c5-3928-4f09-b587-55bedc15e727" (UID: "f22099c5-3928-4f09-b587-55bedc15e727"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:10:00.003115 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.003096 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-istio-envoy" pod "f22099c5-3928-4f09-b587-55bedc15e727" (UID: "f22099c5-3928-4f09-b587-55bedc15e727"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:10:00.003214 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.003193 4951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f22099c5-3928-4f09-b587-55bedc15e727-istio-token" pod "f22099c5-3928-4f09-b587-55bedc15e727" (UID: "f22099c5-3928-4f09-b587-55bedc15e727"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:10:00.101770 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.101727 4951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f22099c5-3928-4f09-b587-55bedc15e727-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:10:00.101770 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.101762 4951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:10:00.101770 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.101776 4951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:10:00.102053 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.101787 4951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:10:00.102053 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.101799 4951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f22099c5-3928-4f09-b587-55bedc15e727-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:10:00.102053 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.101809 4951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f22099c5-3928-4f09-b587-55bedc15e727-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:10:00.102053 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.101822 4951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vghz8\" (UniqueName: \"kubernetes.io/projected/f22099c5-3928-4f09-b587-55bedc15e727-kube-api-access-vghz8\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 08:10:00.622897 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.622854 4951 generic.go:356] "Generic (PLEG): container finished" podID="f22099c5-3928-4f09-b587-55bedc15e727" containerID="71179fdc9d969c06970e3b5bc6f4517d98daacb5570c0bdd3ce9d68a84f05de2" exitCode=0 Oct 30 08:10:00.623125 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.622904 4951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" Oct 30 08:10:00.623125 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.622939 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" event={"ID":"f22099c5-3928-4f09-b587-55bedc15e727","Type":"ContainerDied","Data":"71179fdc9d969c06970e3b5bc6f4517d98daacb5570c0bdd3ce9d68a84f05de2"} Oct 30 08:10:00.623125 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.623010 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc" event={"ID":"f22099c5-3928-4f09-b587-55bedc15e727","Type":"ContainerDied","Data":"60e2b323ef530ab5c13b73a60e0ca19153980f081a810343d1de89c07d09ed22"} Oct 30 08:10:00.623125 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.623038 4951 scope.go:122] "RemoveContainer" containerID="71179fdc9d969c06970e3b5bc6f4517d98daacb5570c0bdd3ce9d68a84f05de2" Oct 30 08:10:00.633168 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.633144 4951 scope.go:122] "RemoveContainer" containerID="71179fdc9d969c06970e3b5bc6f4517d98daacb5570c0bdd3ce9d68a84f05de2" Oct 30 08:10:00.633475 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:10:00.633455 4951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71179fdc9d969c06970e3b5bc6f4517d98daacb5570c0bdd3ce9d68a84f05de2\": container with ID starting with 71179fdc9d969c06970e3b5bc6f4517d98daacb5570c0bdd3ce9d68a84f05de2 not found: ID does not exist" containerID="71179fdc9d969c06970e3b5bc6f4517d98daacb5570c0bdd3ce9d68a84f05de2" Oct 30 08:10:00.633532 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.633487 4951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71179fdc9d969c06970e3b5bc6f4517d98daacb5570c0bdd3ce9d68a84f05de2"} err="failed to get container status \"71179fdc9d969c06970e3b5bc6f4517d98daacb5570c0bdd3ce9d68a84f05de2\": rpc error: code = NotFound desc = could not find container \"71179fdc9d969c06970e3b5bc6f4517d98daacb5570c0bdd3ce9d68a84f05de2\": container with ID starting with 71179fdc9d969c06970e3b5bc6f4517d98daacb5570c0bdd3ce9d68a84f05de2 not found: ID does not exist" Oct 30 08:10:00.689131 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.689086 4951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc"] Oct 30 08:10:00.701693 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:00.701662 4951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-f6clc"] Oct 30 08:10:01.770590 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:01.770541 4951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f22099c5-3928-4f09-b587-55bedc15e727" path="/var/lib/kubelet/pods/f22099c5-3928-4f09-b587-55bedc15e727/volumes" Oct 30 08:10:11.662201 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:11.662170 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-795446ff4c-nbl2l_483fcce9-07d5-4224-bfc9-6ae15c087276/manager/0.log" Oct 30 08:10:14.624433 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:14.624392 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-f785bd78b-t42sc_9eea3e53-4722-493d-8820-27d28c7e5270/maas-api/0.log" Oct 30 08:10:20.457989 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:20.457957 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-nk2g9_454b78a6-5a90-41a9-8ff5-fa16edf2b257/manager/0.log" Oct 30 08:10:22.203414 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:22.203380 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cf9d6756f-sxt26_9c0b8673-ee48-45f2-8ea6-ac112d33cd51/kube-auth-proxy/0.log" Oct 30 08:10:22.403797 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:22.403759 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-67v85_9287b67d-925d-46da-aa33-a5f47b3b7a1e/istio-proxy/0.log" Oct 30 08:10:22.784121 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:22.784086 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-b2nrg_6c9e850f-e9fb-4aa6-ba8d-4113648338e3/istio-proxy/0.log" Oct 30 08:10:23.022804 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:10:23.022767 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 08:10:23.403642 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:23.403607 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-m4l2r_c834492c-b73f-41d5-b3c0-95ad02f5e005/payload-pre-processing/0.log" Oct 30 08:10:23.599179 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:23.599147 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-pbml8_861854e2-21ba-40a6-b1b5-e3e36bd0858f/payload-processing/0.log" Oct 30 08:10:23.803951 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:23.803906 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7f6bb8cb6f-fzpwq_0c49a4a4-033f-4a62-8bc4-6b59531e8bf7/router/0.log" Oct 30 08:10:24.809620 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:24.809565 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-m8ksd_c9b6e389-2e4a-45d1-a3cf-a15d930ebb63/main/0.log" Oct 30 08:10:25.191907 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:25.191836 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-7dtn7_2cb568ba-79ac-405d-af09-bde6c2b47a53/main/0.log" Oct 30 08:10:25.609262 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:25.609234 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-xnj7x_4ec98545-1b89-4b01-883f-098acce664e7/main/0.log" Oct 30 08:10:25.804693 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:25.804647 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-7g49k_2a4b5a5d-94d1-47d0-9bbe-3849812d8ece/main/0.log" Oct 30 08:10:36.319233 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319196 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-x6pbf/must-gather-2xkwj"] Oct 30 08:10:36.319757 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319594 4951 cpu_manager.go:414] "containerMap: removing container" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" Oct 30 08:10:36.319757 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319605 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" Oct 30 08:10:36.319757 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319616 4951 cpu_manager.go:414] "containerMap: removing container" podUID="2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985" containerName="maas-api" Oct 30 08:10:36.319757 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319622 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985" containerName="maas-api" Oct 30 08:10:36.319757 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319632 4951 cpu_manager.go:414] "containerMap: removing container" podUID="85d10cb3-066c-46fa-a149-209a92a409df" containerName="istio-proxy" Oct 30 08:10:36.319757 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319637 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85d10cb3-066c-46fa-a149-209a92a409df" containerName="istio-proxy" Oct 30 08:10:36.319757 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319645 4951 cpu_manager.go:414] "containerMap: removing container" podUID="9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3" containerName="istio-proxy" Oct 30 08:10:36.319757 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319650 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3" containerName="istio-proxy" Oct 30 08:10:36.319757 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319660 4951 cpu_manager.go:414] "containerMap: removing container" podUID="75cf7bf0-7eb7-4f6c-a5a1-c9436acec579" containerName="maas-api" Oct 30 08:10:36.319757 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319664 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75cf7bf0-7eb7-4f6c-a5a1-c9436acec579" containerName="maas-api" Oct 30 08:10:36.319757 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319670 4951 cpu_manager.go:414] "containerMap: removing container" podUID="db4df589-43fd-4413-8c51-de66dd1ec2e2" containerName="istio-proxy" Oct 30 08:10:36.319757 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319675 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db4df589-43fd-4413-8c51-de66dd1ec2e2" containerName="istio-proxy" Oct 30 08:10:36.319757 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319684 4951 cpu_manager.go:414] "containerMap: removing container" podUID="8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc" containerName="payload-processing" Oct 30 08:10:36.319757 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319688 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc" containerName="payload-processing" Oct 30 08:10:36.319757 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319695 4951 cpu_manager.go:414] "containerMap: removing container" podUID="9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3" containerName="istio-proxy" Oct 30 08:10:36.319757 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319699 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3" containerName="istio-proxy" Oct 30 08:10:36.319757 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319710 4951 cpu_manager.go:414] "containerMap: removing container" podUID="2149f8b0-9707-4e35-851e-af05b7842bf3" containerName="maas-api" Oct 30 08:10:36.319757 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319714 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2149f8b0-9707-4e35-851e-af05b7842bf3" containerName="maas-api" Oct 30 08:10:36.319757 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319726 4951 cpu_manager.go:414] "containerMap: removing container" podUID="3ad4d41c-7af3-453d-9005-d34264ed3acd" containerName="istio-proxy" Oct 30 08:10:36.319757 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319731 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ad4d41c-7af3-453d-9005-d34264ed3acd" containerName="istio-proxy" Oct 30 08:10:36.319757 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319737 4951 cpu_manager.go:414] "containerMap: removing container" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" Oct 30 08:10:36.319757 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319742 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" Oct 30 08:10:36.319757 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319747 4951 cpu_manager.go:414] "containerMap: removing container" podUID="64ba04d1-bbfe-4ed3-b0b2-55d554a025fb" containerName="payload-pre-processing" Oct 30 08:10:36.319757 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319752 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ba04d1-bbfe-4ed3-b0b2-55d554a025fb" containerName="payload-pre-processing" Oct 30 08:10:36.319757 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319758 4951 cpu_manager.go:414] "containerMap: removing container" podUID="b08b669b-c0a7-462c-9b63-aabbb0254dc1" containerName="istio-proxy" Oct 30 08:10:36.319757 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319763 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b08b669b-c0a7-462c-9b63-aabbb0254dc1" containerName="istio-proxy" Oct 30 08:10:36.319757 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319769 4951 cpu_manager.go:414] "containerMap: removing container" podUID="92a44e19-26dc-46b2-8a2b-5d5a33941087" containerName="maas-api" Oct 30 08:10:36.319757 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319774 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92a44e19-26dc-46b2-8a2b-5d5a33941087" containerName="maas-api" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319784 4951 cpu_manager.go:414] "containerMap: removing container" podUID="3f265353-a1b5-4361-bce3-65723e30beec" containerName="maas-api" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319789 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f265353-a1b5-4361-bce3-65723e30beec" containerName="maas-api" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319795 4951 cpu_manager.go:414] "containerMap: removing container" podUID="33977737-ec70-47e5-a689-4c471c77c42e" containerName="payload-processing" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319800 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33977737-ec70-47e5-a689-4c471c77c42e" containerName="payload-processing" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319808 4951 cpu_manager.go:414] "containerMap: removing container" podUID="51ace3b8-b759-4dd3-bc27-be16dcdc9d97" containerName="payload-processing" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319812 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51ace3b8-b759-4dd3-bc27-be16dcdc9d97" containerName="payload-processing" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319818 4951 cpu_manager.go:414] "containerMap: removing container" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319823 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319829 4951 cpu_manager.go:414] "containerMap: removing container" podUID="1016f833-117b-4e7a-9678-5dc7b3656bfe" containerName="payload-pre-processing" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319834 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1016f833-117b-4e7a-9678-5dc7b3656bfe" containerName="payload-pre-processing" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319841 4951 cpu_manager.go:414] "containerMap: removing container" podUID="798d3036-be81-460a-819d-2385a159ecb1" containerName="payload-processing" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319845 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="798d3036-be81-460a-819d-2385a159ecb1" containerName="payload-processing" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319852 4951 cpu_manager.go:414] "containerMap: removing container" podUID="3ad4d41c-7af3-453d-9005-d34264ed3acd" containerName="istio-proxy" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319856 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ad4d41c-7af3-453d-9005-d34264ed3acd" containerName="istio-proxy" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319863 4951 cpu_manager.go:414] "containerMap: removing container" podUID="dc75560d-af65-4bad-9bd4-c7e7156882be" containerName="maas-api" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319867 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc75560d-af65-4bad-9bd4-c7e7156882be" containerName="maas-api" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319872 4951 cpu_manager.go:414] "containerMap: removing container" podUID="325e2932-31f7-49a9-bac7-d8ac85a4500f" containerName="payload-processing" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319877 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="325e2932-31f7-49a9-bac7-d8ac85a4500f" containerName="payload-processing" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319882 4951 cpu_manager.go:414] "containerMap: removing container" podUID="f22099c5-3928-4f09-b587-55bedc15e727" containerName="istio-proxy" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319888 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f22099c5-3928-4f09-b587-55bedc15e727" containerName="istio-proxy" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319894 4951 cpu_manager.go:414] "containerMap: removing container" podUID="147db729-cd3c-40f4-ae95-dd4db5869e60" containerName="maas-api" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319898 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="147db729-cd3c-40f4-ae95-dd4db5869e60" containerName="maas-api" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319906 4951 cpu_manager.go:414] "containerMap: removing container" podUID="334b282b-4bb9-4ebf-a675-b6940059cdd6" containerName="test-curl-7602-92594c" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319913 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="334b282b-4bb9-4ebf-a675-b6940059cdd6" containerName="test-curl-7602-92594c" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319923 4951 cpu_manager.go:414] "containerMap: removing container" podUID="3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b" containerName="istio-proxy" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319930 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b" containerName="istio-proxy" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319938 4951 cpu_manager.go:414] "containerMap: removing container" podUID="b08b669b-c0a7-462c-9b63-aabbb0254dc1" containerName="istio-proxy" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319943 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b08b669b-c0a7-462c-9b63-aabbb0254dc1" containerName="istio-proxy" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319955 4951 cpu_manager.go:414] "containerMap: removing container" podUID="910285f4-8b96-44a7-a0e5-542f04538151" containerName="maas-api" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319960 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="910285f4-8b96-44a7-a0e5-542f04538151" containerName="maas-api" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319967 4951 cpu_manager.go:414] "containerMap: removing container" podUID="3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b" containerName="istio-proxy" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319971 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b" containerName="istio-proxy" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319982 4951 cpu_manager.go:414] "containerMap: removing container" podUID="8f3ce93c-f668-4afc-b693-bfb8f0ad8428" containerName="istio-proxy" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319988 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f3ce93c-f668-4afc-b693-bfb8f0ad8428" containerName="istio-proxy" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319994 4951 cpu_manager.go:414] "containerMap: removing container" podUID="4b05487e-0050-4d5a-8875-4878d2b565a4" containerName="payload-pre-processing" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.319999 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b05487e-0050-4d5a-8875-4878d2b565a4" containerName="payload-pre-processing" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320005 4951 cpu_manager.go:414] "containerMap: removing container" podUID="070057c0-00c3-49dc-a4d2-868885c1685c" containerName="payload-pre-processing" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320010 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="070057c0-00c3-49dc-a4d2-868885c1685c" containerName="payload-pre-processing" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320018 4951 cpu_manager.go:414] "containerMap: removing container" podUID="a796d7df-446c-419e-be91-f479e9bfa8e2" containerName="istio-proxy" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320022 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a796d7df-446c-419e-be91-f479e9bfa8e2" containerName="istio-proxy" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320102 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="db4df589-43fd-4413-8c51-de66dd1ec2e2" containerName="istio-proxy" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320114 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="51ace3b8-b759-4dd3-bc27-be16dcdc9d97" containerName="payload-processing" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320121 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c39d5b-24e5-433d-a50f-b7e5f45b993f" containerName="main" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320128 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="1016f833-117b-4e7a-9678-5dc7b3656bfe" containerName="payload-pre-processing" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320133 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="798d3036-be81-460a-819d-2385a159ecb1" containerName="payload-processing" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320138 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ad4d41c-7af3-453d-9005-d34264ed3acd" containerName="istio-proxy" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320144 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc75560d-af65-4bad-9bd4-c7e7156882be" containerName="maas-api" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320150 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="325e2932-31f7-49a9-bac7-d8ac85a4500f" containerName="payload-processing" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320154 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="f22099c5-3928-4f09-b587-55bedc15e727" containerName="istio-proxy" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320158 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="147db729-cd3c-40f4-ae95-dd4db5869e60" containerName="maas-api" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320163 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="334b282b-4bb9-4ebf-a675-b6940059cdd6" containerName="test-curl-7602-92594c" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320167 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b" containerName="istio-proxy" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320173 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="b08b669b-c0a7-462c-9b63-aabbb0254dc1" containerName="istio-proxy" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320178 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="910285f4-8b96-44a7-a0e5-542f04538151" containerName="maas-api" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320182 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fe6f39a-7b8b-4e1b-aa0e-f25ba292e31b" containerName="istio-proxy" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320188 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f3ce93c-f668-4afc-b693-bfb8f0ad8428" containerName="istio-proxy" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320194 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b05487e-0050-4d5a-8875-4878d2b565a4" containerName="payload-pre-processing" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320198 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="070057c0-00c3-49dc-a4d2-868885c1685c" containerName="payload-pre-processing" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320204 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="a796d7df-446c-419e-be91-f479e9bfa8e2" containerName="istio-proxy" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320209 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb929374-4eb9-4a81-b81a-b89098b9fb20" containerName="main" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320214 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb0ca62-df89-4bd5-bd6a-5be2b3ccb985" containerName="maas-api" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320220 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="85d10cb3-066c-46fa-a149-209a92a409df" containerName="istio-proxy" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320225 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3" containerName="istio-proxy" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320267 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="75cf7bf0-7eb7-4f6c-a5a1-c9436acec579" containerName="maas-api" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320277 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cb399dc-ec0a-4ed7-8ef2-fed66142c3cc" containerName="payload-processing" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320293 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="2149f8b0-9707-4e35-851e-af05b7842bf3" containerName="maas-api" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320300 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fcbe6d6-ae98-48cf-b62f-c14e0f7f3106" containerName="main" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320306 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ba04d1-bbfe-4ed3-b0b2-55d554a025fb" containerName="payload-pre-processing" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320311 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="92a44e19-26dc-46b2-8a2b-5d5a33941087" containerName="maas-api" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320316 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f265353-a1b5-4361-bce3-65723e30beec" containerName="maas-api" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320321 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="33977737-ec70-47e5-a689-4c471c77c42e" containerName="payload-processing" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320390 4951 cpu_manager.go:414] "containerMap: removing container" podUID="db4df589-43fd-4413-8c51-de66dd1ec2e2" containerName="istio-proxy" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320396 4951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db4df589-43fd-4413-8c51-de66dd1ec2e2" containerName="istio-proxy" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320467 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b29cf13-7e6a-42d3-9a18-bcd9e9fe48e3" containerName="istio-proxy" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320476 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="db4df589-43fd-4413-8c51-de66dd1ec2e2" containerName="istio-proxy" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320481 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ad4d41c-7af3-453d-9005-d34264ed3acd" containerName="istio-proxy" Oct 30 08:10:36.320739 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.320488 4951 memory_manager.go:365] "RemoveStaleState removing state" podUID="b08b669b-c0a7-462c-9b63-aabbb0254dc1" containerName="istio-proxy" Oct 30 08:10:36.323624 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.323606 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x6pbf/must-gather-2xkwj" Oct 30 08:10:36.348110 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.348080 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x6pbf/must-gather-2xkwj"] Oct 30 08:10:36.358802 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.358769 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-x6pbf\"/\"default-dockercfg-kpcw8\"" Oct 30 08:10:36.359032 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.358827 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x6pbf\"/\"kube-root-ca.crt\"" Oct 30 08:10:36.359032 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.358826 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x6pbf\"/\"openshift-service-ca.crt\"" Oct 30 08:10:36.416117 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.416082 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dq844/must-gather-kcr7x"] Oct 30 08:10:36.419770 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.419747 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dq844/must-gather-kcr7x" Oct 30 08:10:36.424155 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.424129 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngx2k\" (UniqueName: \"kubernetes.io/projected/f9f5cf3b-15c1-49fa-ad82-13e6a8ee8a65-kube-api-access-ngx2k\") pod \"must-gather-2xkwj\" (UID: \"f9f5cf3b-15c1-49fa-ad82-13e6a8ee8a65\") " pod="openshift-must-gather-x6pbf/must-gather-2xkwj" Oct 30 08:10:36.424302 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.424200 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f9f5cf3b-15c1-49fa-ad82-13e6a8ee8a65-must-gather-output\") pod \"must-gather-2xkwj\" (UID: \"f9f5cf3b-15c1-49fa-ad82-13e6a8ee8a65\") " pod="openshift-must-gather-x6pbf/must-gather-2xkwj" Oct 30 08:10:36.442343 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.442292 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dq844\"/\"kube-root-ca.crt\"" Oct 30 08:10:36.442892 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.442867 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dq844\"/\"openshift-service-ca.crt\"" Oct 30 08:10:36.444542 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.444513 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dq844/must-gather-kcr7x"] Oct 30 08:10:36.446063 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.446039 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dq844\"/\"default-dockercfg-jlpw5\"" Oct 30 08:10:36.525116 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.525074 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aa14ee27-e1b9-4af0-9d53-e1fa88f5fc69-must-gather-output\") pod \"must-gather-kcr7x\" (UID: \"aa14ee27-e1b9-4af0-9d53-e1fa88f5fc69\") " pod="openshift-must-gather-dq844/must-gather-kcr7x" Oct 30 08:10:36.525285 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.525185 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f9f5cf3b-15c1-49fa-ad82-13e6a8ee8a65-must-gather-output\") pod \"must-gather-2xkwj\" (UID: \"f9f5cf3b-15c1-49fa-ad82-13e6a8ee8a65\") " pod="openshift-must-gather-x6pbf/must-gather-2xkwj" Oct 30 08:10:36.525285 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.525238 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh9hg\" (UniqueName: \"kubernetes.io/projected/aa14ee27-e1b9-4af0-9d53-e1fa88f5fc69-kube-api-access-sh9hg\") pod \"must-gather-kcr7x\" (UID: \"aa14ee27-e1b9-4af0-9d53-e1fa88f5fc69\") " pod="openshift-must-gather-dq844/must-gather-kcr7x" Oct 30 08:10:36.525285 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.525265 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngx2k\" (UniqueName: \"kubernetes.io/projected/f9f5cf3b-15c1-49fa-ad82-13e6a8ee8a65-kube-api-access-ngx2k\") pod \"must-gather-2xkwj\" (UID: \"f9f5cf3b-15c1-49fa-ad82-13e6a8ee8a65\") " pod="openshift-must-gather-x6pbf/must-gather-2xkwj" Oct 30 08:10:36.525629 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.525609 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f9f5cf3b-15c1-49fa-ad82-13e6a8ee8a65-must-gather-output\") pod \"must-gather-2xkwj\" (UID: \"f9f5cf3b-15c1-49fa-ad82-13e6a8ee8a65\") " pod="openshift-must-gather-x6pbf/must-gather-2xkwj" Oct 30 08:10:36.551788 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.551757 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngx2k\" (UniqueName: \"kubernetes.io/projected/f9f5cf3b-15c1-49fa-ad82-13e6a8ee8a65-kube-api-access-ngx2k\") pod \"must-gather-2xkwj\" (UID: \"f9f5cf3b-15c1-49fa-ad82-13e6a8ee8a65\") " pod="openshift-must-gather-x6pbf/must-gather-2xkwj" Oct 30 08:10:36.625949 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.625852 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh9hg\" (UniqueName: \"kubernetes.io/projected/aa14ee27-e1b9-4af0-9d53-e1fa88f5fc69-kube-api-access-sh9hg\") pod \"must-gather-kcr7x\" (UID: \"aa14ee27-e1b9-4af0-9d53-e1fa88f5fc69\") " pod="openshift-must-gather-dq844/must-gather-kcr7x" Oct 30 08:10:36.626118 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.626016 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aa14ee27-e1b9-4af0-9d53-e1fa88f5fc69-must-gather-output\") pod \"must-gather-kcr7x\" (UID: \"aa14ee27-e1b9-4af0-9d53-e1fa88f5fc69\") " pod="openshift-must-gather-dq844/must-gather-kcr7x" Oct 30 08:10:36.626351 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.626336 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aa14ee27-e1b9-4af0-9d53-e1fa88f5fc69-must-gather-output\") pod \"must-gather-kcr7x\" (UID: \"aa14ee27-e1b9-4af0-9d53-e1fa88f5fc69\") " pod="openshift-must-gather-dq844/must-gather-kcr7x" Oct 30 08:10:36.653544 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.653502 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh9hg\" (UniqueName: \"kubernetes.io/projected/aa14ee27-e1b9-4af0-9d53-e1fa88f5fc69-kube-api-access-sh9hg\") pod \"must-gather-kcr7x\" (UID: \"aa14ee27-e1b9-4af0-9d53-e1fa88f5fc69\") " pod="openshift-must-gather-dq844/must-gather-kcr7x" Oct 30 08:10:36.658437 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.658415 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x6pbf/must-gather-2xkwj" Oct 30 08:10:36.733745 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.733710 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dq844/must-gather-kcr7x" Oct 30 08:10:36.953403 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.953376 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x6pbf/must-gather-2xkwj"] Oct 30 08:10:36.953673 ip-10-0-132-173 kubenswrapper[4951]: W1030 08:10:36.953642 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9f5cf3b_15c1_49fa_ad82_13e6a8ee8a65.slice/crio-5c6c06c3c14e2cd4765301a3a6b4afd5442e05089808e2ce223d8afc9142221c WatchSource:0}: Error finding container 5c6c06c3c14e2cd4765301a3a6b4afd5442e05089808e2ce223d8afc9142221c: Status 404 returned error can't find the container with id 5c6c06c3c14e2cd4765301a3a6b4afd5442e05089808e2ce223d8afc9142221c Oct 30 08:10:36.955532 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:36.955514 4951 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 08:10:37.047610 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:37.047566 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dq844/must-gather-kcr7x"] Oct 30 08:10:37.048394 ip-10-0-132-173 kubenswrapper[4951]: W1030 08:10:37.048369 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa14ee27_e1b9_4af0_9d53_e1fa88f5fc69.slice/crio-8d98545451543053ed1c2f3d29d4213f497718199cbf2b58db3de890a44c4062 WatchSource:0}: Error finding container 8d98545451543053ed1c2f3d29d4213f497718199cbf2b58db3de890a44c4062: Status 404 returned error can't find the container with id 8d98545451543053ed1c2f3d29d4213f497718199cbf2b58db3de890a44c4062 Oct 30 08:10:37.797972 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:37.797926 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dq844/must-gather-kcr7x" event={"ID":"aa14ee27-e1b9-4af0-9d53-e1fa88f5fc69","Type":"ContainerStarted","Data":"8d98545451543053ed1c2f3d29d4213f497718199cbf2b58db3de890a44c4062"} Oct 30 08:10:37.799183 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:37.799153 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x6pbf/must-gather-2xkwj" event={"ID":"f9f5cf3b-15c1-49fa-ad82-13e6a8ee8a65","Type":"ContainerStarted","Data":"5c6c06c3c14e2cd4765301a3a6b4afd5442e05089808e2ce223d8afc9142221c"} Oct 30 08:10:41.542473 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:41.542433 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ts77g_96024683-dd4e-4d78-b5ad-6aa9ccc4bc05/global-pull-secret-syncer/0.log" Oct 30 08:10:41.743742 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:41.743713 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-5bl7s_8f93e69e-f48e-4c0a-ab3d-655739b6a8ae/connection-checker/0.log" Oct 30 08:10:41.882026 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:41.881986 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-7dwjw_14b28600-2ad3-47ff-a157-bcfacc6ab2ff/connection-checker/0.log" Oct 30 08:10:42.020902 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:42.020871 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-zwfbr_239195cf-db90-4e49-9064-8f7aa4af9b3a/connection-checker/0.log" Oct 30 08:10:42.546725 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:42.546693 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mjwrd_5225f97b-21b7-453c-8d19-ccfce5ee6d54/konnectivity-agent/0.log" Oct 30 08:10:42.819653 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:42.819627 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-173.ec2.internal_5b1dc20b8a7bd35e96a3f207cff8defc/haproxy/0.log" Oct 30 08:10:42.827817 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:42.827777 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dq844/must-gather-kcr7x" event={"ID":"aa14ee27-e1b9-4af0-9d53-e1fa88f5fc69","Type":"ContainerStarted","Data":"0fed8076ba56fa46d1aefaebb5294e799f25592cb56c5ce3c3885f2d45d5b442"} Oct 30 08:10:42.827985 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:42.827821 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dq844/must-gather-kcr7x" event={"ID":"aa14ee27-e1b9-4af0-9d53-e1fa88f5fc69","Type":"ContainerStarted","Data":"b614fe746669f6a8095a191ae4254f559369c8ea47778222342efba7a755f433"} Oct 30 08:10:42.829662 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:42.829638 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x6pbf/must-gather-2xkwj" event={"ID":"f9f5cf3b-15c1-49fa-ad82-13e6a8ee8a65","Type":"ContainerStarted","Data":"5bda7908286a0b7331ced9a95ad8ecaafba67ec154aa8bbdc8563168476e37da"} Oct 30 08:10:42.829766 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:42.829670 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x6pbf/must-gather-2xkwj" event={"ID":"f9f5cf3b-15c1-49fa-ad82-13e6a8ee8a65","Type":"ContainerStarted","Data":"f22de21cd891e9c0b0a94a2ca65aa92337410c2f5023b165a65a46953c967a24"} Oct 30 08:10:42.873122 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:42.873059 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-dq844/must-gather-kcr7x" podStartSLOduration=1.6062325670000002 podStartE2EDuration="6.873039923s" totalImagesPullingTime="5.266807356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:10:42.870030813 +0000 UTC m=+4449.213078780" watchObservedRunningTime="2026-10-30 08:10:42.873039923 +0000 UTC m=+4449.216087892" Oct 30 08:10:42.911992 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:42.911942 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-x6pbf/must-gather-2xkwj" podStartSLOduration=2.026764419 podStartE2EDuration="6.91192959s" totalImagesPullingTime="4.885165171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:10:42.91149227 +0000 UTC m=+4449.254540237" watchObservedRunningTime="2026-10-30 08:10:42.91192959 +0000 UTC m=+4449.254977549" Oct 30 08:10:43.844012 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:43.843922 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-795446ff4c-nbl2l_483fcce9-07d5-4224-bfc9-6ae15c087276/manager/0.log" Oct 30 08:10:44.369331 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:44.369298 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-795446ff4c-nbl2l_483fcce9-07d5-4224-bfc9-6ae15c087276/manager/0.log" Oct 30 08:10:48.558084 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:48.557990 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-795446ff4c-nbl2l_483fcce9-07d5-4224-bfc9-6ae15c087276/manager/0.log" Oct 30 08:10:49.018437 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:49.018408 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-795446ff4c-nbl2l_483fcce9-07d5-4224-bfc9-6ae15c087276/manager/0.log" Oct 30 08:10:50.154831 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:50.154795 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-nk2g9_454b78a6-5a90-41a9-8ff5-fa16edf2b257/manager/0.log" Oct 30 08:10:51.195193 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:51.195154 4951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k6rrn/perf-node-gather-daemonset-rtl2t"] Oct 30 08:10:51.200316 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:51.200291 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-rtl2t" Oct 30 08:10:51.217787 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:51.217744 4951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-k6rrn\"/\"default-dockercfg-lq6ls\"" Oct 30 08:10:51.221072 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:51.221041 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k6rrn/perf-node-gather-daemonset-rtl2t"] Oct 30 08:10:51.229856 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:51.229833 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k6rrn\"/\"kube-root-ca.crt\"" Oct 30 08:10:51.277631 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:51.277566 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfbab9a2-e106-4738-adf7-d7a0c10a9441-lib-modules\") pod \"perf-node-gather-daemonset-rtl2t\" (UID: \"bfbab9a2-e106-4738-adf7-d7a0c10a9441\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-rtl2t" Oct 30 08:10:51.277788 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:51.277677 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv5pf\" (UniqueName: \"kubernetes.io/projected/bfbab9a2-e106-4738-adf7-d7a0c10a9441-kube-api-access-jv5pf\") pod \"perf-node-gather-daemonset-rtl2t\" (UID: \"bfbab9a2-e106-4738-adf7-d7a0c10a9441\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-rtl2t" Oct 30 08:10:51.277788 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:51.277770 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bfbab9a2-e106-4738-adf7-d7a0c10a9441-podres\") pod \"perf-node-gather-daemonset-rtl2t\" (UID: \"bfbab9a2-e106-4738-adf7-d7a0c10a9441\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-rtl2t" Oct 30 08:10:51.277926 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:51.277902 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfbab9a2-e106-4738-adf7-d7a0c10a9441-sys\") pod \"perf-node-gather-daemonset-rtl2t\" (UID: \"bfbab9a2-e106-4738-adf7-d7a0c10a9441\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-rtl2t" Oct 30 08:10:51.278016 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:51.277997 4951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bfbab9a2-e106-4738-adf7-d7a0c10a9441-proc\") pod \"perf-node-gather-daemonset-rtl2t\" (UID: \"bfbab9a2-e106-4738-adf7-d7a0c10a9441\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-rtl2t" Oct 30 08:10:51.320810 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:51.320786 4951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k6rrn\"/\"openshift-service-ca.crt\"" Oct 30 08:10:51.379348 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:51.379316 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bfbab9a2-e106-4738-adf7-d7a0c10a9441-podres\") pod \"perf-node-gather-daemonset-rtl2t\" (UID: \"bfbab9a2-e106-4738-adf7-d7a0c10a9441\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-rtl2t" Oct 30 08:10:51.379504 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:51.379355 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfbab9a2-e106-4738-adf7-d7a0c10a9441-sys\") pod \"perf-node-gather-daemonset-rtl2t\" (UID: \"bfbab9a2-e106-4738-adf7-d7a0c10a9441\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-rtl2t" Oct 30 08:10:51.379504 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:51.379396 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bfbab9a2-e106-4738-adf7-d7a0c10a9441-proc\") pod \"perf-node-gather-daemonset-rtl2t\" (UID: \"bfbab9a2-e106-4738-adf7-d7a0c10a9441\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-rtl2t" Oct 30 08:10:51.379504 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:51.379468 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfbab9a2-e106-4738-adf7-d7a0c10a9441-sys\") pod \"perf-node-gather-daemonset-rtl2t\" (UID: \"bfbab9a2-e106-4738-adf7-d7a0c10a9441\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-rtl2t" Oct 30 08:10:51.379504 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:51.379468 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bfbab9a2-e106-4738-adf7-d7a0c10a9441-podres\") pod \"perf-node-gather-daemonset-rtl2t\" (UID: \"bfbab9a2-e106-4738-adf7-d7a0c10a9441\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-rtl2t" Oct 30 08:10:51.379504 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:51.379492 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bfbab9a2-e106-4738-adf7-d7a0c10a9441-proc\") pod \"perf-node-gather-daemonset-rtl2t\" (UID: \"bfbab9a2-e106-4738-adf7-d7a0c10a9441\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-rtl2t" Oct 30 08:10:51.379752 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:51.379556 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfbab9a2-e106-4738-adf7-d7a0c10a9441-lib-modules\") pod \"perf-node-gather-daemonset-rtl2t\" (UID: \"bfbab9a2-e106-4738-adf7-d7a0c10a9441\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-rtl2t" Oct 30 08:10:51.379752 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:51.379622 4951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv5pf\" (UniqueName: \"kubernetes.io/projected/bfbab9a2-e106-4738-adf7-d7a0c10a9441-kube-api-access-jv5pf\") pod \"perf-node-gather-daemonset-rtl2t\" (UID: \"bfbab9a2-e106-4738-adf7-d7a0c10a9441\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-rtl2t" Oct 30 08:10:51.379752 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:51.379685 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfbab9a2-e106-4738-adf7-d7a0c10a9441-lib-modules\") pod \"perf-node-gather-daemonset-rtl2t\" (UID: \"bfbab9a2-e106-4738-adf7-d7a0c10a9441\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-rtl2t" Oct 30 08:10:51.407087 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:51.407063 4951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv5pf\" (UniqueName: \"kubernetes.io/projected/bfbab9a2-e106-4738-adf7-d7a0c10a9441-kube-api-access-jv5pf\") pod \"perf-node-gather-daemonset-rtl2t\" (UID: \"bfbab9a2-e106-4738-adf7-d7a0c10a9441\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-rtl2t" Oct 30 08:10:51.514302 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:51.514268 4951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-rtl2t" Oct 30 08:10:51.819356 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:51.817896 4951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k6rrn/perf-node-gather-daemonset-rtl2t"] Oct 30 08:10:51.821897 ip-10-0-132-173 kubenswrapper[4951]: W1030 08:10:51.821865 4951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbfbab9a2_e106_4738_adf7_d7a0c10a9441.slice/crio-be0c8690cc4fc159e9021e87bc01df06eed777c9b90d42ed35e138e363a86cdc WatchSource:0}: Error finding container be0c8690cc4fc159e9021e87bc01df06eed777c9b90d42ed35e138e363a86cdc: Status 404 returned error can't find the container with id be0c8690cc4fc159e9021e87bc01df06eed777c9b90d42ed35e138e363a86cdc Oct 30 08:10:51.883522 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:51.883488 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-rtl2t" event={"ID":"bfbab9a2-e106-4738-adf7-d7a0c10a9441","Type":"ContainerStarted","Data":"be0c8690cc4fc159e9021e87bc01df06eed777c9b90d42ed35e138e363a86cdc"} Oct 30 08:10:52.889382 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:52.889340 4951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-rtl2t" event={"ID":"bfbab9a2-e106-4738-adf7-d7a0c10a9441","Type":"ContainerStarted","Data":"a22d90573f4364ad3be5e85c44ddd459b12bf1e38b33897183d9756344a0ebd5"} Oct 30 08:10:52.889835 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:52.889470 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-rtl2t" Oct 30 08:10:52.983708 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:52.983676 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-795446ff4c-nbl2l_483fcce9-07d5-4224-bfc9-6ae15c087276/manager/0.log" Oct 30 08:10:53.022666 ip-10-0-132-173 kubenswrapper[4951]: E1030 08:10:53.022632 4951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r5bpp" containerName="extractor" Oct 30 08:10:53.164398 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:53.164312 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-l6qcv_8ffc7ac6-2db9-4391-8fd5-5e78ba03a4c2/cluster-monitoring-operator/0.log" Oct 30 08:10:53.711741 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:53.711708 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-795446ff4c-nbl2l_483fcce9-07d5-4224-bfc9-6ae15c087276/manager/0.log" Oct 30 08:10:54.682399 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:54.682368 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q7nb2_f800ff9e-dc3a-4d35-a111-93f13e27e594/node-exporter/0.log" Oct 30 08:10:54.810285 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:54.810258 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q7nb2_f800ff9e-dc3a-4d35-a111-93f13e27e594/kube-rbac-proxy/0.log" Oct 30 08:10:54.935630 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:54.935530 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q7nb2_f800ff9e-dc3a-4d35-a111-93f13e27e594/init-textfile/0.log" Oct 30 08:10:58.526637 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:58.526608 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cf9d6756f-sxt26_9c0b8673-ee48-45f2-8ea6-ac112d33cd51/kube-auth-proxy/0.log" Oct 30 08:10:58.596721 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:58.596686 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-67v85_9287b67d-925d-46da-aa33-a5f47b3b7a1e/istio-proxy/0.log" Oct 30 08:10:58.664705 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:58.664671 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-b2nrg_6c9e850f-e9fb-4aa6-ba8d-4113648338e3/istio-proxy/0.log" Oct 30 08:10:58.864561 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:58.864494 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-m4l2r_c834492c-b73f-41d5-b3c0-95ad02f5e005/payload-pre-processing/0.log" Oct 30 08:10:58.905803 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:58.905776 4951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-rtl2t" Oct 30 08:10:58.929650 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:58.929622 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-pbml8_861854e2-21ba-40a6-b1b5-e3e36bd0858f/payload-processing/0.log" Oct 30 08:10:58.947308 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:58.947251 4951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-rtl2t" podStartSLOduration=7.947238939 podStartE2EDuration="7.947238939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:10:52.932194613 +0000 UTC m=+4459.275242578" watchObservedRunningTime="2026-10-30 08:10:58.947238939 +0000 UTC m=+4465.290286904" Oct 30 08:10:59.004550 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:59.004519 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7f6bb8cb6f-fzpwq_0c49a4a4-033f-4a62-8bc4-6b59531e8bf7/router/0.log" Oct 30 08:10:59.082719 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:59.082672 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-cjn9t_5943f8dc-3e86-49fa-ac40-70a1f11fb69e/networking-console-plugin/0.log" Oct 30 08:10:59.302548 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:59.302516 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cf9d6756f-sxt26_9c0b8673-ee48-45f2-8ea6-ac112d33cd51/kube-auth-proxy/0.log" Oct 30 08:10:59.371322 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:59.371289 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-67v85_9287b67d-925d-46da-aa33-a5f47b3b7a1e/istio-proxy/0.log" Oct 30 08:10:59.440005 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:59.439978 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-b2nrg_6c9e850f-e9fb-4aa6-ba8d-4113648338e3/istio-proxy/0.log" Oct 30 08:10:59.632520 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:59.632434 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-m4l2r_c834492c-b73f-41d5-b3c0-95ad02f5e005/payload-pre-processing/0.log" Oct 30 08:10:59.697346 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:59.697311 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-pbml8_861854e2-21ba-40a6-b1b5-e3e36bd0858f/payload-processing/0.log" Oct 30 08:10:59.774238 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:10:59.774203 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7f6bb8cb6f-fzpwq_0c49a4a4-033f-4a62-8bc4-6b59531e8bf7/router/0.log" Oct 30 08:11:00.229582 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:11:00.229540 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/3.log" Oct 30 08:11:00.233851 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:11:00.233815 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zw4km_42802760-36a8-4b79-8690-5a00528503c7/console-operator/4.log" Oct 30 08:11:02.612394 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:11:02.612360 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-z5nm2_584e4eb5-6e5e-4f96-b97a-eb58fd762347/volume-data-source-validator/0.log" Oct 30 08:11:04.287274 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:11:04.287241 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-nk2g9_454b78a6-5a90-41a9-8ff5-fa16edf2b257/manager/0.log" Oct 30 08:11:05.156740 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:11:05.156707 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-nk2g9_454b78a6-5a90-41a9-8ff5-fa16edf2b257/manager/0.log" Oct 30 08:11:05.587445 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:11:05.587411 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vgrqz_d930accb-e4d8-404b-8155-1b6a8edd812e/dns/0.log" Oct 30 08:11:05.590909 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:11:05.590886 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cf9d6756f-sxt26_9c0b8673-ee48-45f2-8ea6-ac112d33cd51/kube-auth-proxy/0.log" Oct 30 08:11:05.655510 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:11:05.655482 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-67v85_9287b67d-925d-46da-aa33-a5f47b3b7a1e/istio-proxy/0.log" Oct 30 08:11:05.708050 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:11:05.708021 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vgrqz_d930accb-e4d8-404b-8155-1b6a8edd812e/kube-rbac-proxy/0.log" Oct 30 08:11:05.720107 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:11:05.720071 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-b2nrg_6c9e850f-e9fb-4aa6-ba8d-4113648338e3/istio-proxy/0.log" Oct 30 08:11:05.836115 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:11:05.836087 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qq6f5_4664d551-2cbf-42c5-a730-0d8c088f4f0f/dns-node-resolver/0.log" Oct 30 08:11:05.903711 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:11:05.903614 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-m4l2r_c834492c-b73f-41d5-b3c0-95ad02f5e005/payload-pre-processing/0.log" Oct 30 08:11:05.966244 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:11:05.966214 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-pbml8_861854e2-21ba-40a6-b1b5-e3e36bd0858f/payload-processing/0.log" Oct 30 08:11:06.038249 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:11:06.038212 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7f6bb8cb6f-fzpwq_0c49a4a4-033f-4a62-8bc4-6b59531e8bf7/router/0.log" Oct 30 08:11:07.365425 ip-10-0-132-173 kubenswrapper[4951]: I1030 08:11:07.365393 4951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9fpvq_d1b84f62-7f09-42d0-8ed1-a2447efa3bf3/node-ca/0.log"